Methodology

One of the more crucial parts of our job at Pickr as technology journalists is the need to recreate the same-same experience between test cycles.

As such, Pickr has specific methodologies for various products as detailed on this page, because while an experience is always going to be subjective, it is also backed up by the same approaches to indicate that reviews and analysis aren’t just a dice throw in the dark, and also aren’t just a casual first impression (except where otherwise indicated).

It does need to be noted, however, that the methodology found in this section is only applicable to reviews published by Pickr.

Websites tracked by Pickr’s tracking system are subject to their own methodologies.

Mobile (phones and smartphones)

Phones and smartphones can take anywhere between two days and six days to review, but usually take closer to three or four.

Typically, a smartphone review is based around the general use-case scenario all phones are made for: day-to-day use.

Because we can’t simulate what a day is exactly, Pickr’s test team takes it through the course of their day.

This means the phone is charged throughout the night and taken off charge at 6am, with the course of the day tested with streaming audio, web surfing, email browsing, picture taking, and even a few phone calls here and there. The phone is also setup and connected to no less than two push-based email accounts, allowing work and home to be accounted for in the email world, and testing how much strain the push service has on the battery.

Connection tests are also performed for the handset, allowing speed tests to not just simulate a strain on the device, but also demonstrate actual real world speeds that are attainable. Unless otherwise noted, all speed tests are based in Sydney’s CBD and North Sydney (Australia), and both on Telstra’s 4GX network.

Audio on day one is handled exclusively with wired headphones, allowing a picture to be painted of how well the phone handles wired versus wireless technology, and throughout this time, battery percentages are captured at random intervals.

The battery is pushed to its limits through that regular day, which shows us what the battery will do under normal situations, with some battery tests pushing over into the second or third day.

When that first test is done, we do it again for what is labelled as the second test day (even if it might already be the second or third day), and this time we switch on wireless devices, connecting the phone to Bluetooth headphones and at least one smart device, like a smartband or a smartwatch.

After both of these major test days are completed, we continue using the device until the review is completed, usually in the second day of methodology to see if it is consistent.

Audio (headphones, speakers)

Audio products such as headphones, earphones, in-ear monitors, speakers, soundbars, and generally anything else that transmits sound has to be reviewed based on a combination of technical description and subjective listening.

Reviewing always has the downside of it being totally subjective, though Pickr’s reviewer likes to think the program he set up at his former workplace of GadgetGuy allows the process to be a little more transparent thanks to its consistency.

As such, Pickr’s audio reviews are handled by a sound test that you can listen to yourself using either Apple Music, Google Play Music, or Spotify.

The playlist we use is one stored in lossless audio, much of which is 24-bit, allowing the reviewer to hear the best source of sound that is possible, whether or not you agree with the virtues of 24-bit high-resolution audio.

For the 2016 to 2017 period, Pickr’s sound test can be heard using the following playlist.

Also available on…

listen-google-play

High resolution audio files for the Pickr/Listnr playlist appear as follows:

  • Balmorhea – Candor (Botany Remix) (16-bit, 44kHz)
  • Pentatonix – Lean On (24-bit, 44kHz)
  • Ellie Goulding – Army (24-bit, 44kHz)
  • Propellerheaders – History Repeating (16-bit, 44kHz)
  • Mark Ronson – Uptown Funk (24-bit, 88kHz)
  • Michael Jackson – Billie Jean (24-bit, 176kHz)
  • Marvin Gaye – Ain’t No Mountain High Enough (24-bit, 96kHz)
  • Maroon 5 – Sugar (24-bit, 96kHz)
  • Sia – Elastic Heart (24-bit, 44kHz)
  • Beck – Dreams (24-bit, 96kHz)
  • Phil Collins – In The Air Tonight (24-bit, 192kHz)
  • Muse – Madness (24-bit, 96kHz)
  • Deftones – Change (In the House of Flies) (24-bit, 96kHz)
  • Nirvana – Come As You Are (24-bit, 96kHz)
  • The Who – Baba O’Riley (24-bit, 96kHz)
  • Beach Boys – God Only Knows (24-bit, 192kHz)
  • Ben Folds – Yes Man (24-bit, 96kHz)
  • Nigel Kennedy – Walk on the Ice (16-bit, 44kHz)
  • Christian McBride – Fat Bach and Greens (24-bit, 96kHz)
  • Etta James – At Last (24-bit, 192kHz)
  • Dave Brubeck – Take Five (24-bit, 176kHz)
  • Miles Davis – Flamenco Sketches (24-bit, 192kHz)
  • Kurt Elling – Bonita Cuba (24-bit, 96kHz)
  • Norah Jones – Come Away With Me (24-bit, 192kHz)
  • Radiohead – Daydreaming (24-bit, 48kHz)