Connect with us


The Future of Personalised Learning



Technology-enabled personalized learning systems continue to capture the imaginations of educators, and for good reason. The promise of tailored learning for every student, providing them just what they need when they need it to master a concept at just the right pace and with just the right kind of help to do so all at a massive scale is tantalizing.

Personalized learning refers to instruction in which the pace of learning and the instructional approach are optimized for the needs of each learner. Learning objectives, instructional approaches, and instructional content (and its sequencing) may all vary based on learner needs. In addition, learning activities are made available that are meaningful and relevant to learners, driven by their interests and often self-initiated. (NETP 2016)

But realizing this vision is incredibly complicated. There are so many variables to measure and control for, so many inferences to judge and decision points to set, that the prospect becomes immediately overwhelming. The natural instinct of many designers of these systems has been to reduce the variables as much as possible, thus exerting greater control over the environment and the learner, with the hope of increasing the reliability of outcomes.

Design Continuum: Closed vs Open

In doing so, designers move to one end of a design continuum. That continuum runs from closed and controlled on the one hand vs open and context aware on the other. The closed, controlled approach reminds me of a virtual reality headset. It encloses the learner in the system, isolating them, and attempts to provide for their every need, without referencing a larger context. It reserves most of the control of the system for itself.

In contrast, I think of an open, context aware approach as an augmented reality (AR) display found on modern fighter jets and more and more in the automotive industry. This approach overlays critical digital information on top of the outside world by projecting it onto the cockpit window or the windshield of a car. It provides the pilot or driver constant information synchronized with the changing external conditions. More recently, AR is showing up in the entertainment industry and even in classrooms. It situates and supports the user in a broader context with the goal to prompt and interpret, but not to control the learner.

Closed Personalized Learning Systems

The closed approach to design characterizes the vast majority of the first wave of personalized learning systems in schools and most of the intelligent tutoring systems that preceded them. As a result, most personalized learning systems tend to:

  • Be stand alone solutions
  • Focus on a single subject area
  • Run independently of other data systems
  • Run independently of the teacher’s input and control
  • Limit the learners agency in decision making about their learning

They are algorithm centric; that is, they assume the algorithm is going to teach the student in the best and fastest way possible. If based on sound research from the learning sciences, it may very well be that the algorithm will outperform more naive approaches. But systems architected in this manner may also have significant disadvantages:

  • They tend to isolate the student from peers and teachers
  • They tend to bog down when a student is not just a little bit off, but a lot off (when Hint #3 and Alternative Explanation #5 still doesn’t do the trick)
  • They provide a siloed dataset that relates only to itself and which tends to be reported only within the system
  • They don’t benefit from knowledge that teachers have about students from other contexts or from insight that students might have about themselves
  • They are generally unaware of learning experiences outside of the system that might have changed the mastery level and/or learning needs of students
  • They are entirely dependent upon the itemset within the system to build their adaptations.

Like virtual reality headsets, they create a learning context around the student and immerse them in it. This provides tremendous control to the system designer to eliminate distractions, which can be beneficial to learners who might otherwise become overwhelmed. But it also blocks out other, potentially helpful, sources of input.

Context Aware Personalized Learning Systems

Personalized learning systems could be more powerful, more useful, more relevant, and more accurate if they reoriented from being algorithm centric to being context aware. This approach still runs on algorithms, of course, but, unlike a closed system, a context-aware system is open to other sources of external input that may be able to provide crucial information. These systems tend to leave a significant degree of decision-making up to the user of the system.

Context aware personalized learning systems would have the following characteristics:

1. Be aware of other points of input and use them to inform its algorithm.

a. They would gather information from external assessments; from other learning systems; from teachers, coaches, mentors, and peers; and from available background information on student’s strengths and needs. They would be hungry for data from outside sources and would use this to troubleshoot and adjust their own assertions and predictions about the learner.

2. Be aware of other points of output and allow data to flow out in machine readable formats.

a. They would feed their output data into broader teacher dashboards and enable a view of the whole student by seamlessly contributing their data as one piece of a larger puzzle. They would provide data about learners in formats useful to researchers, administrators, and school counselors, all according to laws and best practices around privacy and security.

3. Be aware of teaching and learning that happens outside of the system.

a. They would include a mechanism to allow a teacher (or the learners themselves) to input that the learner had practiced or mastered material externally to the system. Of course, the system could still provide a brief quiz, test, or activity to verify the accuracy of the report.b. They would include mechanisms that would also allow a teacher to report potential regression, perhaps after a long absence or summer break. The system would then be alerted to follow up to verify that previously mastered concepts were still intact and remediate as needed.c. More advanced systems could also enable and track learning outside of the system and account for it within the system. For example, they could suggest activities outside of the system that the student is either ready for or that the student needs extra help with. After the external activity takes place, they incorporate the results, verifying mastery as needed.

4. Be aware of and responsive to input from external experts, such as teachers and researchers.

a. They would recognize external learning experts and be appropriately responsive to their input. For example, the system might make broader adjustments to its algorithms, increasing or decreasing difficulty levels based on expert input. Of course, the system could still verify this input with additional measurement and could present any discrepancies between the expert’s assessment and the systems assessment back to the expert for consideration.

5. Be aware of and responsive to input from the student.

a. While learners can be unreliable sources of information about their own mastery, systems that collect feedback from them could benefit from understanding their perceived efficacy, progress, and emotional state, all of which impacts learning and performance. Students could share when they were feeling like the system was going too fast or too slow or to cry “Help!” when they were stuck somewhere and the system was not recognizing their distress. Students could also indicate their level of effort or confidence in their responses. This feedback could be used to adjust to their frame of mind and also could be used to improve the system.b. They would allow the learner to have meaningful agency in their learning. This approach would allow the user to make substantial choices (not just choosing whether to do addition first or subtraction first) that allow them to tailor content to their interests and needs. This approach would work to maximize student choice while still providing sophisticated customization of learning.

Of course, these systems also have drawbacks. As alluded to above, student (and teacher) input can be subjective, incomplete, or just plain wrong. That is why systems designed in this way also need safeguards built in to validate the information coming from external sources.

First Steps

A context aware personalized learning system will require significant change. It will certainly require a paradigm shift among many who design these systems. It will require robust and meaningful data interoperability standards among personalized learning systems at a level of granularity that does not presently exist. It also will require a great deal of new engineering and functionality. It is hard to imagine these kinds of systems being fully functional anytime soon. However, we can start now by creating a foundation for this functionality in our current systems by:

  • Complying with interoperable data standards
  • Creating APIs for both receiving and sharing data with other systems
  • Designing systems capable of processing learning data from external activities and assessments
  • Designing algorithms to evaluate external inputs and adapt accordingly
  • Maximizing and prioritizing features that support learning agency

In short, this approach would open up closed systems to make them responsive to other systems and responsive to teachers, learners, and others outside of the system and what they might do to impact learning. More sophisticated implementations would become a partner with the those outside the system in adjusting the algorithms inside the system while also verifying that the changes are justified.

A Crossroads: Virtual Reality or Augmented Reality

We are at a crossroads in our design approach to personalized learning. We can keep burrowing in, creating an ever narrower and more controlled environment, like a VR world that grows ever more elaborate and detailed, yet exists for and isolates a single person inside an enclosed headset. Or we can turn outward to create context aware systems that function like augmented reality displays that sense, interpret, and add value to the wider world of learning, leaving as much agency in the hands of the learner as possible. The path to context aware systems is longer and more difficult, but ultimately may lead to more meaningful learning and engagement that, in my view, is worth the effort.


About the Author

This article was written by Joseph South is Director of the Office of Educational Technology at the U.S. Department of Education.


Building Yelp: A History Lesson



In the fall of 2004, Jeremy Stoppelman caught the flu.

He had just arrived in San Francisco that summer, so he jumped online in hopes of finding a recommendation for a doctor. Instead, all Stoppelman found were bare bones directories and useless information.

But this gave him an idea. He and Russel Simmons were in San Francisco working for a business incubator called MRL Ventures, searching for “the next big thing” on the internet. He met with Simmons over lunch.

The two were in the office of their boss, Max Levchin, pitching their new concept before dinnertime. They didn’t have a PowerPoint presentation or a specific revenue plan; just a sense that they could make something that would appeal to lots of people.

Early photo of Simmons (left) and Stoppelman (right).

Levchin hesitated. “I wasn’t sure if it would work. But the guys were really enthusiastic about it. And in my experience, when you have smart people who work well together, it’s foolish not to invest.”

Maybe he was feeling lucky because it was his 29th birthday, or maybe it was those tens of millions laying around from his recent exit from PayPal, but Levchin agreed. He invested $1 million in the half-baked idea and Stoppelman and Simmons got to work.

Yelp 1.0

So what were they building? The two founders realized from Stoppelman’s doctor experience that the best way to find a business was through word of mouth. But word of mouth hadn’t moved to the web yet. The question they were asking was, “How do we bring those in-person recommendations online?”

They thought the answer was email and that’s exactly what the earliest version of Yelp was. On the website Simmons put together, users could email their friends asking for recommendations on specific locations or types of places. Responses were logged on a communal site for everyone to see.

Their first review came in on October 12, 2004. Katherine W. gave Truly Mediterranean four stars and a simple, but convincing:

“dirt cheap, good falafels.”

Despite that promising review, their idea was a flop. It attracted few users beyond the founders’ friends and family and failed to impress the venture capital investors whom Stoppelman pitched at the end of 2004.

“We got the doors slammed in our face over and over again,” Stoppelman said. Things were starting to fizzle right before their eyes.

The Epiphany

Undeterred, the founders searched for a way to improve their product. One day, they noticed something.

The site had a link, buried somewhere in the footer, that you could click if you wanted to submit a review without being asked. While poring over their analytics, they realized that people were not only finding that link, they were beginning to use it — often.

They watched as users submitted unsolicited reviews more and more. It got even bigger than the email-requested reviews. People would write 5, 10, or 15 reviews in one sitting.

They knew they had stumbled upon something big. So in February 2005, the duo launched a brand new site, this time focused entirely on unsolicited reviews. Yelp 2.0 saw an immediate rise in traffic. It was a hit.

The Foundation

A 2005 version of

To kick-start the process of building a platform for this new review system, they purchased a database of over 20 million business locations. This database was old and inaccurate, but it created the framework for what Yelp called “claimed business locations”.

The empty business pages functioned as an open invitation for people to submit reviews. It motivated people to, at the very least, write a few words about the business. In fact, many of the early reviews were just that: “this place is great”, or “this place sucked.” But as time passed, reviewers started to take the platform more seriously and write longer, deeper reviews. The framework paid off in dividends later.

Also, they didn’t subordinate the user’s contributions to professional reviews, as on Citysearch, or to directory information, like yellow-pages sites. Instead, Yelp motivated people to share reviews through praise and attention , something no one else was doing. Those social networking features were what made them stand out.

Getting Social

Now that they had the right direction, they needed to grow their user base. Without the cash for a national rollout, Stoppelman decided to first focus on making Yelp famous locally.

With the help of a buzz-marketing guru he hired on a whim, Stoppelman decided to select a few dozen people — the most active reviewers on the site — and throw them an open-bar party. As a joke, he called the group the Yelp Elite Squad.

A Yelp Elite event

Levchin thought the idea was crazy: “I was like, ‘Holy crap, we’re nowhere near profitability; this is ridiculous,’ “. But 100 people showed up to the first party, and traffic to the site began to increase. Since the parties were reserved for prolific reviewers, they gave casual users a reason to use the site more and nonusers a reason to join.

By June 2005, Yelp had 12,000 reviewers, most of them in the Bay Area. In November, Stoppelman went back to the VCs and bagged $5 million from Bessemer Venture Partners. He used the money to throw more parties and hire party planners — Yelp called them “community managers” — in New York, Chicago, and Boston. Community managers and the Yelp Elite Squad still exist today.


The number of reviewers on the site grew to 100,000 by 2006. Stoppelman also raised several million more in venture capital. By summer 2006, Yelp had one million monthly visitors and they were slowly adding more cities.

Now that user counts were growing, they focused on their next problem: they needed to get merchants to play a much deeper role. A growing user base of reviewers was wonderful, but the other side of the coin was the businesses themselves. Not to mention they were Yelp’s only source of revenue.

They decided to begin an aggressive drive to get merchants to claim business listings, populate them (e.g. menus, hours, website, etc), and motivate their own customers to review their experiences on Yelp.

The Yelp sticker

One of the ways they did this was by using a sticker. It was a genius move.

Most businesses were already familiar with Zagat and Mobile stickers and the impact they had on awareness. But Yelp was more aggressive with it and even handed out extra marketing materials. This had a remarkable effect on the review count. Organic review counts shot up and more businesses got on board.

Yelp stickers became almost ubiquitous at famous restaurants in the Bay Area and continue to serve the company today. They stand as a daily reminder of Yelp to the potential reviewer, the potential visitor, and the merchant.

Yelp’s Legacy

Stoppelman, Simmons, and the rest of the Yelp team were persistent, humble enough to pivot, and savvy enough to see the real problems they faced and to use creative methods to overcome them.

Yelp continued to grow. The service kept adding cities and eventually went international. They launched a successful mobile app. Stoppelman gathered tens of millions in venture capital and then took it IPO. As of this writing, the company has a market cap of almost $3.7 billion.

They’ve made a few mistakes along the way, and some say they’re in the middle of a process of disruption. But Yelp — the original king of place reviews — spawned a score of apps and startups and changed the way consumers view their relationship with businesses.

For that, they get…

About the Author
This article was written by Jordan Bowman of
Continue Reading


Top 10 KPIs for your Tech Startup



Key Performance Indicators (KPIs) are data points that measure your company’s performance. They help you answer specific business questions such as:

  • Is the business financially viable?
  • What is working well and what needs to be improved?
  • What is driving customers to purchase your products?
  • How can the company improve its profitability?

Let’s look at 10 KPIs that are useful to track for ecommerce, marketplace and SaaS based businesses.

Monthly Active Users (MAU): This demonstrates how many users visit your platform, website or service every month and are “active”. That could mean they engage with the blog, click on the pricing page or interact with the contact form. This is done by the number of users that visit your platform in a certain time period (albeit monthly in this case). For an ecommerce company like Drover, a peer to peer marketplace for leasing cars to drivers, seeing the MAU increasing means it is attracting new users to the platform.

Conversion Rate (CR): According to BigCommerce, even if you are doing everything right –  you’d only expect to make a sale only 2% of the time. Clearly there are ecommerce companies that exceed that, however. Conversion rate can be calculated easily, as per the below:

Conversion Rate: # of sales / # of visitors

Provider to Consumer Ratio: This is important when tracking the growth of a marketplace business. This is defined as the number of customers a single provider on the supply side of the marketplace can serve. This varies radically across different marketplace businesses, according to Phil hu: AirBnB 70:1, Uber 50:1 and eBay 5:1.

Average Order Value (AoV): AoV is crucial to determine how much revenue you can generate over time. This describes the average size of an order on your platform. Naturally, the higher the average order size the better.

Customer Acquisition Cost (CAC): This is a single most important metric that runs across most business models. The amount it costs to acquire a new customer. Ideally, your CAC should be zero – that is every new customer is referred by another potential customer or customer base grows organically. However, that is rarely possible. To bring new customers onto the platform, you’ll have to spend some money. It is really important to track this over time, and see by how much you’re able to decrease it.

Customer Lifetime Value (CLV): This represents the total amount of revenue that you expect to get from each customer. To calculate the CLV, it depends on how long the customer is retained on the platform, how many repeat purchases does the customer make and what is the average order size. To get a general idea, CLV can be calculated based on the average order value multiplied by the average number of repeat purchases per customer.

Churn Rate: This metric measures the number of customers your platform loses over a given period of time (daily/monthly/annually). Churn rate is critical for SaaS based businesses, where customers pay subscription recurring payments. If the churn rate is high, clearly it means the customer base is unhappy.

Monthly Revenue Rate (MRR): MRR describes the predictable revenue stream of your platform. To calculate MRR, you need to understand the total number of customers per month, and know how much revenue does each customer creates, as per the below:

MRR = Amount of revenue per customer * Total # of customers

Contribution Margin (CM): Contribution margin is the margin that is left after you deduct all variable costs of producing the product or service from the total revenues. A common mistake entrepreneurs make is lumping fixed costs of building a product or service and variable costs together and deduct that from the revenue to understand profit. However, fixed costs remain permanent in the business, no matter how many products or service you produce, it is the variable costs (such marketing spend) that you can change in your business. Some of the most common ways to use the CM is to understand which products or services to continue building and which ones to kill or how to price the products or services.

Net Promoter Score (NPS): This is a great metric to understand if customers are likely to refer you to other users such that your platform can grow organically. To calculate NPS you must ask a specific question:

How likely is it that you would recommend [brand] to a friend of colleague?

And ask the user to answer with a number between 0-10. You can read more information about tracking NPS in this guide here.

Customers that give you a 6 or below are Detractors, a score of 7 or 8 are called Passives, and a 9 or 10 are Promoters.

To calculate your Net Promoter Score, detract the percentage of Detractors from the percentage of Promoters. It is that simple. So, if 50% of respondents were Promoters and 10% were Detractors, your Net Promoter is a score of 40.


About the Author

This article was written by  of the  Path Forward. The Path Forward was developed by Forward Partners, a VC platform that invests in the best ideas and brilliant people. Forward Partners devised The Path Forward to help their founders validate their ideas, build a product, achieve traction, hire a team and raise follow on funding all in the space of 12 months. The Path Forward is a fantastic startup framework for you to utilise as an early stage founder or operator. The framework clearly defines startup creation as being comprised of three steps. The first step of this framework involves understanding customer’s needs.

Continue Reading