Give me a chance here, I’m a layman with a limited understanding of how the internet works.

On a small scale, STW forum doesn’t host pictures because that would use up a lot of memory which they would have to pay for, so all pictures on this forum are hosted elsewhere and linked to or embedded.
Even so, old threads don’t get deleted, so STW must be buying more memory to keep them forever, even if text does use a lot less memory than pictures.

On a much bigger scale, Facebook doesn’t delete old stuff to make way for the new stuff, it’s all there, text and pictures, for ever.
Presumably other similar sites like Twitter and Instagram do the same.
What about Google with their cache of every page on the web?
Is it all backed up or mirrored on more than one location, actually doubling the amount of memory needed?

So, are all these big companies constantly adding more and more memory?
How big is it? Both in a computer sense, how many Gigabites, and a physical sense, how big a building does it need?
Will they just keep on adding more memory in bigger buildings with bigger power supplies for ever, or are they relying on future technology to allow them to store information at the atomic scale, so it will all fit in one small box?

Not quite the answer to your question, but this makes for an interesting read on a related subject:
https://what-if.xkcd.com/31/

One small box

[video]http://youtu.be/iDbyYGrswtg[/video]

So, are all these big companies constantly adding more and more memory?
How big is it? Both in a computer sense, how many Gigabites, and a physical sense, how big a building does it need?

Yes.

Massive. We have hosting centres the size of aircraft hangers stacked with boxes upon boxes.

How many GB, it’s a figure that’s incomprehensible

I look after just EMEA and AP internal company servers, that’s 1000 windows boxes for a decent company, we have about the same for UNIX, just internal within our organisation. I think worldwide we have 30,000 servers for just the day to day running of the business.

Hosting wise we have in excess of 100,000 host machines to ‘serve’ customer business.

I suspect STW forum could be run on a reasonable dual CPU box with a fast drive array and 12GB of RAM.

Just a guess – can a staff member post the specs?

This

is a small chunk of a large datacenter. It’s a bit more serious!

When you get to this level, you also need good security.

In the industry we call it ‘storage’ because ‘memory’ is RAM, but I’m splitting hairs really.

You can just keep adding more infinitely (well until you run out of room on earth I suppose). Their are Data Centres where you can buy/rent cloud storage but providers like the social media companies will usually have their own.

It’s not really legacy data that uses it all either, text on a page like the sort of stuff STW stores is tiny, you can fit whole books in a few KB – as you’ve hinted at it’s pictures, programmes and video that takes up all the room – but storage seems to be getting cheaper and more compact at a faster rate than new standards like HD and now 4K can gobble it up.

Not infinite no. But sufficient to satisfy demand ? Probably 🙂

‘the internet’ is now one of the biggest and fastest growing energy consumers. It burns more fossil and produces more CO2 than many countries do. In 2012 it was 300 million tons of CO2 per annum and probably more now. But instead of using that fuel for heat, transport and industry that energy is used to store videos of goats and everything that Hora has ever typed.

Weeksy, slightly OT, but who do you work for and where are you based? Strange co-incidence, but 10mins ago I emailed my MD regarding data centres. My closing line to her was “What we could really do with is a real live data centre manager to interrogate, I mean chat with”.

UrbanHiker – Member

Weeksy, slightly OT, but who do you work for and where are you based? Strange co-incidence, but 10mins ago I emailed my MD regarding data centres. My closing line to her was “What we could really do with is a real live data centre manager to interrogate, I mean chat with”.

Not something I really want to discuss in public mate, but feel free to email. However I don’t look after our external facing datacentres for 3rd party use etc, but I’m sure I could speak to and put you in touch with someone who does.

Re the energy point:
I was listening to a podcast recently and the entrepreneur owner of storage facilities was saying he’s almost in the energy business as thats the overwhelming consideration when building new data storage.

Maybe we should operate a Homers brain system of one in one out.

Here you go..

The Bottom Line Data Storage

edit to add linky

Maybe we should operate a Homers brain system of one in one out.

To a great extent we’re doing that massively already

We’ve taken our physical servers down by almost 70% in the past 3-4 years. They now are mainly ‘virtual’ which is massively better and more efficient, although not quite as powerful/good for some Apps/users.

We can have say 2 physical servers now taking up the space that 20-30 used to be needed for.

We’ve taken our physical servers down by almost 70% in the past 3-4 years. They now are mainly ‘virtual’ which is massively better and more efficient, although not quite as powerful/good for some Apps/users.

We can have say 2 physical servers now taking up the space that 20-30 used to be needed for.

as in, a number of ‘servers’ running off one physical machine?

as in, a number of ‘servers’ running off one physical machine?

Pretty much yes

Is there, or will there be, an infinite amount of memory on the internet?

Well the Universe itself probably can’t contain an infinite amount of information:
https://edge.org/conversation/the-computational-universe

‘the internet’ is now one of the biggest and fastest growing energy consumers. It burns more fossil and produces more CO2 than many countries do. In 2012 it was 300 million tons of CO2 per annum and probably more now. But instead of using that fuel for heat, transport and industry that energy is used to store videos of goats and everything that Hora has ever typed.

It takes about 1kJ of energy to do a google search.

A lot of data ends up on tape storage. If I was to bother going back through my (unlimited storage) e-mails far enough eventualy it would run out of hard drive and then a little robot arm goes off into a big warehouse and picks up the tape off the shelf and takes it to a machine. Same for most backup storage, you pay a basic rate and it’ll be recoverable overnight, you pay quite a bit more to have it on an actual hard drive in a server.

Its a good question OP.

I once asked a mate who was a Broadband installer, if he went round a corner and all the Broadband fell out of the back of his van, what would it look like as a heap in the road ?

He struggled with that ….

As an aside, based on current technology there are lots of related problems to consider, power consumption being one the big internet companies are keenly aware of.

When Sun released ZFS (Zettabyte File System, kinda like a really advanced FAT/NTFS file system on your laptop hard disk) there was a quote that came from one of the nerds at Sun to give an idea of how powerful/future-proof ZFS was:

‘If we could implement a physical system with the storage capacity that matches the 128-bit address range of ZFS, that we would “literally evaporate all the oceans on earth”. ‘

Anyway, someone did the sums to investigate further:
Link