Intro

This documents Quikly's analysis of the performance impact of the Quikly shopify app on a shopify store.

Requirements

• To be published in the Shopify App Store, your app must not reduce storefront Lighthouse performance scores by more than 10 points.
• To qualify for Built for Shopify status, your app must not reduce Lighthouse performance scores by more than ten points, and must meet other applicable criteria

Testing methodology

Each page is analyzed as many times as needed for a performance number to repeat three times. Only when a performance is achieved three times is it taken into account, after which the counter for that number restarts. The first five numbers (including if duplicated) are taken into account, as a way to eliminate outliers in the final calculation. The extremes will be mentioned but not taken into account mathematically since they tend to be anomalies. Usually this results in around 50 lighthouse reports for each state.

Storefront App Extension Performance

These are the lighthouse reports for the storefront page on mobile (as per the testing guide). Our app loads a banner and also an animated drawer here. Due to the nature of the drawer animation and the need to fetch graphics, it is within expectations that performance would be slightly reduced compared to the clean install of the shopify storefront.

Performance on a Clean Storefront (Starting Score)

The average performance of a clean storefront hovered in the low eighties, peaked once at 91 and bottomed out at 59. Interesting, it would also very consistsently yield a score of 62 or 63 every third or fourth page load. Maybe a nuance with Shopify caching?

Starting Score = (85 + 62 + 83 + 86 + 88) / 5 = (404 / 5) = 80.8

Performance with our App (Ending Score)

The average performance of the storefront with our app against our test servers hovered in the high seventies. Peaking once at 89 and bottoming out at 52. Noticeably, our app seems to underperform on first load after a fresh install, and performs much better on subsequent pageloads. Our app also performs significantly better on desktop rather than mobile, likely due to our app taking up relatively more visual space on mobile, tending to hover in the high eighties and low nineties versus the clean desktop performance hovering the mid-nineties. The addition of our app also seemed to eliminate the frequency with which the storefront lighthouse performance would cyclically hit the low sixties, as it would do clean.

Ending Score = (78 + 79 + 63 + 74 + 76) / 5 = (370 / 5) = 74

Storefront App Extension Performance Ratio

Performance ratio = (Starting Score / Ending Score) = 74 / 80.8 = 0.915

Furthermore, on average, our app does stay within a 10 point reduction of the lighthouse performance score. That being said, there are definitely improvments to be made here. To begin with, testing against our test API is slower than our live API, so there should be performance improvements available there. In addition to that, once client needs become more apparent after launch, there are a lot of variations with how we can store information within shopify to create more elegant loading states for banners that appear immediately, which should drastically improve performance.

Checkout Extension Performance

The lighthouse on mobile for a clean checkout throws an error: The page did not load any content. Please ensure you keep the browser window in the foreground during the load and try again. So we will be using the desktop for this. All in all, using the eye test and with the knowledge of how lighthouse performance check works, the checkout extension should fare much better –– the loading should be virtually instantaneous since the presence of the intial state is determined not by a fetch but by cart attributes.

Performance on a Clean Checkout (Starting Score)

The performance of a clean checkout stayed between low eighties and high seventies with no real outliers.

Starting Score = (82 + 78 + 74 + 83 + 80) / 5 = (397 / 5) = 79.4

Performance with the Checkout Extension (Ending Score)

As per expectations, our checkout extension performs very well in lighthouse performance analysis. It is not 100% clear as to why the score is elevated to above that of a clean page, but regardless of that, the score is well within acceptable ranges and effectively does not reduce lighthouse performance at all. There were no real outliers during numerous reloads –– the checkout extension is very consistent.

Ending Score = (82 + 83 + 83 + 79 + 81) / 5 = (408 / 5) = 80.6

Storefront Checkout Extension Performance Ratio

Performance ratio = (Starting Score / Ending Score) = 80.6 / 79.4 = 1.015