Use HTTP Archive for library usage statistics
Created by: technopagan
I was intrigued when reading your essential next steps, esp. " to find out which versions of which libraries are most commonly used on websites". Since I'm comfy using HTTP Archive + Google BigQueries I ran some initial queries:
I've compared year-over-year growth of the most common JS libraries I could think of (angular|backbone|ember|react|jquery|underscore|lodash|d3|babylon|three|babel|meteor|polymer|dojo|prototype) and ran the query on HTTP Archive's January 1st runs for 2014, 2015 & 2016. Here's a screenshot of the resulting table:
And here's the SQL query to generate this result from Google Bigqueries:
SELECT
year,
type,
CONCAT(STRING(INTEGER(100 * (COUNT - prev) / prev)),
'%') year_growth,
count,
prev prev_count from(
SELECT
COUNT,
LAG(COUNT) OVER(
PARTITION BY
type
ORDER BY
year ) prev,
type,
year
FROM (
SELECT
type,
COUNT,
year
FROM (
SELECT
REGEXP_EXTRACT(url,
r'(angular|backbone|ember|react|jquery|underscore|lodash|d3|babylon|three|babel|meteor|polymer|dojo|prototype)') type,
COUNT(DISTINCT(pageid)) COUNT,
'2016' year
FROM
[httparchive:runs.2016_01_01_requests]
WHERE
REGEXP_MATCH(url,
r'angular|backbone|ember|react|jquery|underscore|lodash|d3|babylon|three|babel|meteor|polymer|dojo|prototype')
GROUP BY
type),
(
SELECT
REGEXP_EXTRACT(url,
r'(angular|backbone|ember|react|jquery|underscore|lodash|d3|babylon|three|babel|meteor|polymer|dojo|prototype)') type,
COUNT(DISTINCT(pageid)) COUNT,
'2015' year
FROM
[httparchive:runs.2015_01_01_requests]
WHERE
REGEXP_MATCH(url,
r'angular|backbone|ember|react|jquery|underscore|lodash|d3|babylon|three|babel|meteor|polymer|dojo|prototype')
GROUP BY
type),
(
SELECT
REGEXP_EXTRACT(url,
r'(angular|backbone|ember|react|jquery|underscore|lodash|d3|babylon|three|babel|meteor|polymer|dojo|prototype)') type,
COUNT(DISTINCT(pageid)) COUNT,
'2014' year
FROM
[httparchive:runs.2014_01_01_requests]
WHERE
REGEXP_MATCH(url,
r'angular|backbone|ember|react|jquery|underscore|lodash|d3|babylon|three|babel|meteor|polymer|dojo|prototype')
GROUP BY
type),
))
WHERE prev IS NOT NULL
ORDER BY year, type
I hope this helps. Let me know how I can assist further.
Thanks for this great plugin!