Chocolate, for most people, is a bit of a staple. Sugary sweets and cakes might pass you by, but this confectionary is hard to avoid. You'll find stacks of it in your local newsagents and shelves of the stuff in the supermarket. Adverts pop up at bus stops and on our televisions – sometimes, for some reason, represented by a gorilla with a drum kit or a pale kid in a cowboy hat. You'll be hard-pressed to find a dessert menu that doesn't give some sort of nod to chocolate. And even one of our nation's favourite children's books literally teaches us social justice by way of a chaotic chocolate factory tour.

There's no two ways about it: we're obsessed. And with good reason, because this elegant concoction of sugar, cocoa and fat is undeniably delicious. Ask around, though, and you'll realise that many of us probably can't remember our first ever taste of chocolate. I certainly can't. I'm told, however, that I was nine months old, and my inaugural encounter with chocolate also happened to coincide with my first sip of Irn Bru. My mum still hasn't forgiven my aunt for that one.

Family feuds aside, not being able to pinpoint exactly when something this readily available entered your life shouldn't be too surprising. My 78-year-old grandfather can recall the exact moment he tucked into his first Cadbury's Dairy Milk. It was 1945, the tail end of World War II and eight years before sweet rationing came to a definitive end. But we're living in a time of plenty and these days, the only chocolate rationing we might have is a desperate, self-inflicted tax on our personal sugar consumption.

The commercialisation of chocolate has transformed a once-coveted extravagance into an everyday luxury while contributing an estimated £1.1bn to the UK economy, too. The days of the 10p Freddo might be long gone, but that's not to say a bar of fairly decent-tasting chocolate will weigh too heavily on your wallet. You can always count on Cadbury's to 'accidentally' fall into your shopping basket, setting your weekly budget back a hefty £1 for a 100g slab.

Buying mainstream chocolate, of course, comes with its own associated costs (palm oil, anyone?). But it also begs the question, in the face of omnipresent brands churning out cheaper chocolate, how on earth have a growing raft of artisan chocolatiers weathered the storm, some even maintaining – or setting up – pricey bricks-and-mortar sites?

"It isn't possible to sell enough chocolates like ours to pay the rent," says Paul A Young in a moment away from the prep kitchen below his shop on Wardour Street. "Unless you're a massive brand that's now self-funded, like Hotel Chocolat, you've got to be really savvy." Best known for his award-winning extravagant flavour combinations, Young is one of a handful of independent chocolatiers keeping the magic of the artisan chocolate shop alive. He's been hand-crafting chocolates for 13 years, and with three stores under his belt, no one could claim that he isn't pretty savvy himself.

Over the course of a decade and a bit, while Young has been creating Willy Wonka-esque creations – involving everything from cranberry and goose fat caramels to Marmite truffles – he's witnessed the rise and rise of one of the nation's favourite chocolate shops. Hotel Chocolat, founded by Angus Thirlwell and Peter Harris in 1987, which opened its first store in North London in 2004, just two years before Young stepped on to the scene. "They've captured the market really well," Young says. "They've taken over from Thorntons in terms of quality on the high street."

Charbonnel et Walker's outpost in the Royal Arcade on Bond Street has stood for 144 years

That's not much of a surprise. Established in 1911, Thorntons was once the place to turn for gift-worthy chocolate boxes and toffees. But with the rise of cheap supermarket chocolate, it failed to keep up with competitors and, according to an article in The Guardian, reported a loss of £30.8m in 2016. Hotel Chocolat, on the other hand, has managed to establish itself as a premium high street brand, retaining and growing its customer base by constantly evolving its product range from simple slabs to cocoa-infused beauty products, flavour-packed truffles and seasonal goodies. As a result, its multi-million pound empire has now expanded to 81 locations, with eight cafés, two restaurants (one of which you'll know as Rabot 1745 in Borough Market) and a hotel in St Lucia in tow.

So if the demand for luxury chocolate is there, how come Young – and the rest of the artisan chocolate producers in London and beyond – haven't seen this level of success? Well, it seems that when it comes to chocolate shops, there can be a crucial difference: there are those who sell, and those who make.

Charbonnel et Walker is one of the former. Its success could probably rival that of Hotel Chocolat – although it's had far longer to establish itself: founded in 1875, its outpost in the Royal Arcade on Bond Street has stood for 144 years, making it London's oldest chocolate shop. With its Royal warrant, Charbonnel et Walker has the privilege of being chocolatier to the Queen, and in 2017, its turnover was a humble £14.8m. But the truffles in its gold-trimmed boxes weren't made on-site that morning; the brand manufactures its chocolates in a factory in Tunbridge Wells. They're produced on a massive scale and with a long shelf-life in mind. This cuts down on labour costs and reduces waste. It's also the reason you can count on Waitrose to have a few of its dusty-pink boxes of Marc de Champagne truffles on sale next time you forget your in-laws' anniversary on the way to dinner.

They're not alone in this, either: Hotel Chocolat's factory is based in Huntingdon, Cambridgeshire, and the only 'fresh' chocolate you'll probably find on the premises of Prestat – double Royal warrant holder and owner of London's second-oldest chocolate shop – is its unbeatable hot chocolate. As delicious as these are, there's a staggering difference between these brands and producers like Young. "We still don't preserve, we still use seasonal ingredients and we still handmake everything every single day," he says, gesturing behind me to a table ladened with his fanciful truffles. "We've done that from day one and we haven't changed."

This distinction is something known only too well by London's leading chocolatier Chantal Coady. When Coady opened her first Rococo shop at the age of 23, on the Kings Road in Chelsea in 1983, no one was creating fresh, handmade chocolate on the same scale. "Apart from the few leftover from the 18th century or 19th century," she recalls, "chocolate shops weren't really a 'thing'".

Back then, if you wanted a special-occasion box of chocolates you'd take a trip to Selfridges or Fortnum & Mason and queue up at the confectionary counter. There, traditional chocolates from heritage British brands and those shipped in from France and Belgium were sold alongside other confectionery such as Mars Bars. Coady even vividly remembers arranging a Cadbury Milk Tray for Sir Michael Caine in Harrods, where she held down a Saturday job during her time at art school. "The pinnacle of luxury," Coady recalls, "was Leonidas' chocolates, Belgian truffles made of fresh cream and butter that definitely weren't made that day, but supposedly only lasted a week."

On their busiest days, the Rococo team could serve as many as 1,000 customers

Coady, in some way, is responsible for turning all that around. "We just realised that it was something very special we could do." She says. "A fresh truffle is not that complicated to make, and it tastes so much better than anything that was made six months ago." It was this philosophy that led to Rococo's monopoly on designer chocolates in the 1980s and early 1990s. On their busiest days, the team could sell to as many as 1,000 customers, before collapsing in a heap and rising again to do it all again the next day.

It's not quite the same now. For one, Rococo no longer makes its chocolates in store. Instead, production has moved to the top floor of 'Rococo Towers' in South London. But 36 years on, Coady has still got a steady stream of customers wandering into her beautifully dressed chocolate shops.

Coady was the one to give Young his first commission and she's no doubt inspired a fair few more chocolatiers to follow in her footsteps. Much, if not all, of her chocolate is organic and she's committed to using only sustainable, ethically sourced ingredients. Her efforts have even been recognised in the highest of places and in June 2014 she was awarded an OBE for services to chocolate making – something she thought was a "scam" when the letter first came through.

I first came across Rococo while training to be one of Jennifer Earle's Chocolate Ecstasy Tour guides. Yes, that's exactly what you think it is and no, I wasn't very good. Earle founded Chocolate Ecstasy Tours in 2005, as a series of immersive walking tours around London's best chocolate shops. As well as tastings at Pierre Hermé and Artisan du Chocolat, Earle also counts Rococo's Belgravia store as one of the stops on her Chelsea route. "I've watched the interest for high-quality chocolate grow every year," says Earle when I ask her about London's love affair with chocolate shops. "We've gone from having only a handful of artisan chocolatiers in the UK to well over a hundred combined.

"At first, almost no one who came on a tour had heard of salted caramel and the fanciest chocolate most people knew of was Lindt." But all that has since changed, apparently. "I haven't introduced anyone to salted caramel for years now. Many of my guests have already tried 100% chocolate and know of some of the craft chocolate makers." Earle reckons that the enduring appeal of London's chocolate shops has something to do with people's ever-growing willingness "to spend money on better-quality food." Sure, the £1 bars are still on their radar, but that interest in quality is only getting stronger.

View on Instagram

This is something Alain Ducasse – who opened his first UK branch of famed Le Chocolat Alain Ducasse in King's Cross's Coal Drop Yard in October last year – seems to have clocked on to. "Minds are changing quite rapidly these days." he says. "London is one of the most vibrant cities and it's seen an increased interest in artisan chocolate."

Ducasse's unique offering has already proved to be popular in both Paris and Tokyo. "In Tokyo, chocolate is often bought as gifts," he says when I ask him what drives customer interest in each city. "Whereas in Paris there are probably more self-indulgent purchases." But how does London measure up? "Londoners are very much open to novelties," he continues. "And, as for the food scene, the demand, and therefore offering, has rapidly evolved over the last decade. Customers have become more educated and are now looking for speciality products."

Finding something special is almost a guarantee when it comes to London's chocolate shops, but does the arrival of this iconic brand and the rest of London's seemingly thriving chocolatiers signal the end of cheap chocolate's reign? "I wish," says Young back at his Soho store. "But very few people can afford just to buy high-end artisan, single origin and single-bean variety chocolate all the time."

Perhaps not, then. And yet, having experienced chocolate on both sides of the spectrum, I've come to think that London's artisan chocolatiers aren't going away anytime soon. If one thing is absolutely certain, it's that people will never stop buying chocolate. And if that chocolate comes from a growing number of artisan chocolatiers sourcing sustainably to create beautiful, hand-crafted and delicious chocolates? Well, that's definitely a trend I can get behind.