When you are ready to test, log into your account and click “Add voice test.” You can save your test as a draft until you are ready to test. Once you click “Submit for Testing,” it gets submitted to the Operations team for launch, and you can no longer make changes.

Choose the device type

Basic and Premium plans include testing with Alexa. Enterprise and Agency accounts can also test with Google.

Enable SMAPI if you are beta testing an Alexa skill

At the top of the Test Setup page, you can enter your Amazon Developer credentials to establish the connection with Alexa’s Skill Management API (SMAPI) or edit it. 

Before testing, integrate our SDK into your Alexa skill
Use your unique API key and follow the documentation available here to integrate our SDK into your skill's code. Integration should take less than 30 minutes, and using it will let you have a 360-degree view of panelists' interactions with your skill!

Enter the application information 

If you have enabled SMAPI for your Alexa skills, most of this information will auto-populate when you select the skill you want to test. If you don’t have SMAPI enabled, enter the following:

  • Application name 
  • Invocation phrase
  • The test name -- this is a name you select to help you distinguish this from other tests you may run
  • The country 
  • The language model -- most countries have one but some countries, like Canada, may have multiple options. For more information, see our Supported Languages and Countries section. 

Enter the test details

For the Overview, enter a description of the skill and any information people may need in order to prepare for testing.

  • Be concise and compelling!
  • Include information about what the app is designed to do and what types of customers might actually use the app once it is available.
  • If the skill for a regional business, indicate where services will be available.
  • If the app is a prototype and has limited functionality, state that so panelists have the right expectations about what they can do.
  • If the panelist will need anything other than their device to complete testing tasks, include that information. For example, if your app sends cards to their smartphone, tell them to have it close by. 

Example Overview 

With the Red Bank Alexa skill, now banking is easier than ever! Just ask your smart speaker to check your account balance, find your nearest branch, pay bills, or transfer funds. With our one-of-a-kind security features, you can rest assured that your voice banking requests remain secure.

This skill is a prototype so the features are limited to the ones in the test tasks. Please keep your smartphone handy so you can view information the app sends for certain activities. Red Bank currently has locations in North Carolina, South Carolina, and Georgia.

In the Tasks section, enter 1-3 tasks specific items you’d like panelists to complete during their test session. 

  • Keep tasks brief - longer tasks leave more room for confusion.
  • Tasks should all be completed in about 20-30 minutes - this leads to better feedback.
  • Explain things in general terms - remember that panelists have different levels of technical knowledge.

If you want panelists to test a specific flow, you can give very specific tasks. If you’re trying to gauge general usability and user sentiment, you’ll want your tasks to be more broad. Here are some example tasks:

  • Open the app and use it as you naturally would based on the prompts.
  • At some point during the test, ask for help.
  • Say “give me a dinner recipe” and find a recipe you like.
  • Say “call a plumber” and see if you can get a call connected to your smartphone.
  • After using the app, exit, re-enter, and try to resume the game.

Notes are optional and serve to help you remember the purpose of the test. They are only visible to you and the Pulse Labs Operations team.

Enter Participant Criteria

Choose “Use my panelists” to upload a CSV file with email addresses or enter a comma-separated list of emails. You will also need to enter a message to include in your panelists’ test invitation. If you plan to use Pulse Labs Panelists, choose the option on the left then select the demographic fields you’d like us to use when inviting panelists to your test.

For both options you will need to select a name for your panel. This lets us store your demographic selections so you can use them for future tests.

Post-test Questions

Basic - tests with your own panelists

All tests come with 5 standard questions: 

  1.  How would you rate your satisfaction with this voice app overall? (Scale of 1 - very bad to 5 - very good)
  2. Were there any points where you got stuck, confused, or frustrated? Please describe what happened.
  3. Which aspects of this voice app did you most enjoy?
  4. Which aspects of this voice app would you recommend improving?
  5. What additional features would you like to see in future versions of this voice app?

Premium accounts and Basic tests with Pulse Labs Panelists

With these, you can enter five of your own custom questions. We offer the following question formats:

  • Short text - allows panelists to write in their own response. They will be able to write several sentences if they choose. Examples: ‘What similar games have you played?’ ‘What age group do you think this application serves best?’
  • Rating - panelists can select 1 to 5 stars. Examples: ‘How likely are you to recommend this to a friend or family member?’ ‘How likely are you to use this application at least once a week?’ 
  • Yes or no - panelists can answer yes or no.Examples: ‘Had you heard of COMPANY before this test?’ ‘Do you own a car?’ 
  • Multiple choice - you can enter multiple options for panelists to choose from. Panelists can choose one response. Examples: ‘What price would you be willing to pay for additional features?’ ‘Which of these best describes your job?’

Enterprise and Agency accounts

You can provide questions customized to your needs using the four question types listed above. Our operations team can help you craft questions to capture the information you’re looking for, and make sure that the number and type of questions will get you the most information. We've found that when required to answer long lists of questions, panelist feedback value and quality declines.

Did this answer your question?