OT: Thinking out loud....

  • Thread starter Thread starter Jim Hubbard
  • Start date Start date
J

Jim Hubbard

File-sharing sucks for large files.

Even when multiple people are sharing the same file you're after (like on
Kazaa Lite K++) the connections are still slow, vary over time with the, and
most will drop connection before you finish the download.

Is there a particular file-sharing network that is geared more towards
larger files?
 
I have found BitTorrent exceedingly useful. Your success in downloading the
file you want does unfortunately depend on its popularity and how many
people are sharing it.

The download is segmented so that different parts are downloaded from
different people. You will still find that people stop sharing the file but
given time you usually get what you are after.

As I say, not ideal but maybe an alternative.

Cheers,

Gareth
 
But doesn't BitTorrent work from websites? It isn't exactly a P2P sharing
client is it?
 
Ok, so I'm just denser than I originally suspected...but, I downloaded
BitTorrent and TorrentStorm and don't see a search feature like you do in
traditional P2P applicartions.

Where do you get .torrent files? Where do you search for files?

Looks like a closed system. Too bad, they almost got it right.

Any system like this need to be open source (i.e. source available on the
website) and should have a forum available, not just some silly Yahoo Group.

I can do this better.

Wanna see?
 
Jim Hubbard said:
Ok, so I'm just denser than I originally suspected...but, I downloaded
BitTorrent and TorrentStorm and don't see a search feature like you do in
traditional P2P applicartions.

That doesn't mean it's not P2P though. Given the way it works, I don't
think search would really be appropriate.
Where do you get .torrent files? Where do you search for files?

I've basically just seen torrent links on websites (eg Slashdot). You
can search for torrents using Google, of course.
Looks like a closed system. Too bad, they almost got it right.

I don't believe it's particularly closed, actually.
Any system like this need to be open source (i.e. source available on the
website)

The Python source is available on the website. More importantly, the
protocol is specified on the website.
and should have a forum available, not just some silly Yahoo Group.

I'm sure there are plenty of forums in one form or another. I don't
happen to know about them as I haven't needed them - the software's
just worked for me.

By "silly Yahoo Group", do you mean the technical support mailing list?
Just because it's using Yahoo Groups doesn't mean it's not perfectly
usable as a mailing list.
I can do this better.

Wanna see?

If you can end up with a better and more successful system than
BitTorrent, I'm sure you'll be much in demand. I look forward to seeing
it.
 
Jon Skeet said:
That doesn't mean it's not P2P though. Given the way it works, I don't
think search would really be appropriate.

That brings me back to how to find .torrent files. If there is no search,
it seems less user-friendly to me than other file-sharing networks. It also
breaks stride with the users of other file-sharing software. Things should
be kept as simple as possible to ensurte the greates following possible.

(Not that you need to read it, but I have found it to be a great
help...."Don't Make Me Think" by Steve Krug. It's about user thought and
activity patterns as they relate to web page design. Although that
definition makes the book sound a tad technical and dry, it is neither. It
is an easy afternoon read and Steve's ideas can be related to software
design as well.)
I've basically just seen torrent links on websites (eg Slashdot). You
can search for torrents using Google, of course.

Just not as user friendly as other P2P programs. I have seen some sites
online that charge $9.95 to $34.95 for links to .torrent files. That dog
just won't hunt.
I don't believe it's particularly closed, actually.

On second-look, I agree.
The Python source is available on the website. More importantly, the
protocol is specified on the website.

Found it. Don't know Python, but good to know the source is out there.

As for my little test P2P file-sharing app, I think I'll make a DLL
available in .Net to make creating your own client as easy as possible. The
more clients you have, the more popular the protocol will be, the more files
will be shared on the network.
I'm sure there are plenty of forums in one form or another. I don't
happen to know about them as I haven't needed them - the software's
just worked for me.

By "silly Yahoo Group", do you mean the technical support mailing list?
Yes.

Just because it's using Yahoo Groups doesn't mean it's not perfectly
usable as a mailing list.

It is usable, but having to get a YahooID and request to be added to the
group seems like it may turn some people off. A lot of forum software is
free and requires very little (if any) login info for people to use.

Microsoft had a great idea with the whole passport thing, but screwed it up
by charging so much for a small business otr website to use it that
Microsoft Passports are not the login panacea they should have been.
If you can end up with a better and more successful system than
BitTorrent, I'm sure you'll be much in demand. I look forward to seeing
it.

I'll give myself a deadline of December 24, 2004. I am starting from
scratch, so the first BETA will not be pretty. While I'm at it, I'll try
and think of ways that the games, music and movie industries can use it to
distribute trailers, demos and even paid-for items. (The distribution thing
is easy, the paid-for-items thing...not so much.)

I will post here when the beta is ready.
 
Jim Hubbard said:
That brings me back to how to find .torrent files. If there is no search,
it seems less user-friendly to me than other file-sharing networks. It also
breaks stride with the users of other file-sharing software. Things should
be kept as simple as possible to ensurte the greates following possible.

You find the files either by searching with Google, or (in my
experience, certainly) by being in the relevant community which
publishes the torrent files in the first place. For instance, I might
be reading a review of a game, and there's a torrent of a demo
available. No need to search.

Just not as user friendly as other P2P programs. I have seen some sites
online that charge $9.95 to $34.95 for links to .torrent files. That dog
just won't hunt.

So don't pay - I'm sure the *vast* majority of torrent users don't.
There are plenty of free links as well. Would you think less of other
P2P programs if there were sites which charged to do things with them?
The presence of pay sites doesn't remove the presence of non-pay sites.

As for my little test P2P file-sharing app, I think I'll make a DLL
available in .Net to make creating your own client as easy as possible. The
more clients you have, the more popular the protocol will be, the more files
will be shared on the network.

While a .NET DLL will certainly help for some clients, if you *require*
..NET you've just removed a huge bunch of users.
It is usable, but having to get a YahooID and request to be added to the
group seems like it may turn some people off. A lot of forum software is
free and requires very little (if any) login info for people to use.

Yahoo doesn't require significant amounts. It's free and pretty easy -
less time consuming for most people than downloading and installing
..NET, for instance.
Microsoft had a great idea with the whole passport thing, but screwed it up
by charging so much for a small business otr website to use it that
Microsoft Passports are not the login panacea they should have been.

Arguably, making people put a *bit* more effort into getting technical
support isn't a bad idea - it might mean that by the time they ask the
question they're interested in, they have a vested interest in asking
it in a useful way.
I'll give myself a deadline of December 24, 2004. I am starting from
scratch, so the first BETA will not be pretty. While I'm at it, I'll try
and think of ways that the games, music and movie industries can use it to
distribute trailers, demos and even paid-for items. (The distribution thing
is easy, the paid-for-items thing...not so much.)

I will post here when the beta is ready.

Sure. I'll remain cynical about it being a BT-beater until that happens
though... it's not like the world has been short of P2P apps and had to
make do with any old rubbish.
 
That brings me back to how to find .torrent files. If there is no search,
it seems less user-friendly to me than other file-sharing networks. It also
breaks stride with the users of other file-sharing software. Things should
be kept as simple as possible to ensurte the greates following possible.
<snip>

http://suprnova.org

Contains just about any (non-adult) files that are out on the
internet. Careful, though, as the MPAA is definitely aware of the
site.

There are many others. http://btsites.tk/ has links to lots of BT
sites.

(Sorry if this is too OT, as I can't find the original message.)

Austin
 
You find the files either by searching with Google, or (in my
experience, certainly) by being in the relevant community which
publishes the torrent files in the first place. For instance, I might
be reading a review of a game, and there's a torrent of a demo
available. No need to search.

I finally found a definition of BitTorrent that was dumbed down enough for
me to understand why BitTorrent doesn't offer a search engine. I found the
following defitnition of BitTorrent....

"BitTorrent has no network in the sense of KaZaA or Napster - it's a
protocol. People or companies wanting to distribute a file essentially
create their own private P2P network which only consists of whoever is
downloading the file at the time. These miniature networks are formed around
a ``tracker'', which is a server program operated by the entity wishing to
share a file. " (http://www.joestewart.org/p2p.html)

This means that there is no "network" in the traditional P2P sense,
therefore, no searching peers for files.

While a .NET DLL will certainly help for some clients, if you *require*
.NET you've just removed a huge bunch of users.

You are right. But, as the Microsoft worm turns, more and more users will
be able to DL the framework. And, if the .Net framework is available via
the network, a simplified win32 app could DL the .Net framework via the same
file-sharing network.

I'll also make the protocol public so that other people can code clients in
the language of their choice.

Yahoo doesn't require significant amounts. It's free and pretty easy -
less time consuming for most people than downloading and installing
.NET, for instance.

This is something I have posted about also. If Microsoft is really banking
on .Net going forward (and I believe they are) why in the world wouldn't
Microsoft have thought through its distribution better than they did?

The first thing I don't get with the .Net distribution is why the .Net
framework is not a critical update on the Windows Update site.

The second thing I don't get is... The .Net framework has a neat feature
that allows a program run from a network location (intra or inter) to only
download the DLLs it needs as it needs them. Why wasn't this same type of
functionality built into the Windows OS (or even the .Net EXEs or the .Net
setup for .Net applications in Visual Studio) for programs that need the
..Net runtime to run. If it were, simply running a .Net exe would initiate a
download and installation of only the .Net DLLs required to run the .Net
application.

Each PC would then trickle down the .Net framework as it is needed and would
avoid the massive 23+MB download.
Arguably, making people put a *bit* more effort into getting technical
support isn't a bad idea - it might mean that by the time they ask the
question they're interested in, they have a vested interest in asking
it in a useful way.

Technical support aside, the main reason for using the Microsoft Passport
service, as I understood it, was to have a single login for many websites.
Microsoft has charged so mch for the service that this is not an option for
the majority of small businesses on the internet (whihc just happens to be a
majority of all businesses on the internet).

Sure. I'll remain cynical about it being a BT-beater until that happens
though... it's not like the world has been short of P2P apps and had to
make do with any old rubbish.

You should be cynical. I would be if I were in your shoes. And, Lord knows
we have a few too many file-sharing applications with no real open standards
that anyone follows. Also the massive amount of user requests that flow
through the major systems (like Gnutella and Kazaa) seem to slow the systems
down quite a bit.

There are good points to a lot of systems. The distributed server model of
one model has the distributor of the file-sharing software running a main
server that only accepts connections from other mini-servers that are run by
individuals interested in starting a file-sharing server for his/her
particular interests. This allows people to connect to mini-servers that
specialize in the type of file they are looking for and speeds file
searching while increasing the likelihood of finding your file.

The main server simply keeps a log of all mini-servers and their specialties
for the users to choose a mini-server to connect to. Connections to more
than one would be nice, but is not a feature.

Others use hashes to find exact file matches. While this the added benefit
of finding exact matches for a file, this also has a drawback because a file
that is 99% or even 90% the same may suffice or my be able to donate
portions of the file being sought. This also means a boatload of returns
for a search when most of the returns may be basically the same file.
Absolute bit precision is not usually needed in something like an MP3 or a
movie trailer.

Another nice feature would be automatically disconnecting from a file server
if the download time exceeds a certain, user-supplied limit. Each user
could also have a ranking that indicates the number of downloads
initated/completed, the same for uploads and the average time online. This
may help to choose a more reliable file-server and should result in
increased functionality for the more reliable file servers.

And, why should you have to tell a file-sharing application to "find new
sources" if it knows it can't dl the file from the original file-server
(because it is no longer online or whatever)? Shouldn't it just search and
find the file automatically (perhaps fail after X tries or Y minutes without
a match)?

The biggest change could come in the way we share and retrieve files.
Current methods usually result in many partial downloads because of
file-servers going on- and off-line. Since the very nature of a peer file
server is transient, this should be taken into consideration when designing
the system. Thus far, (IMHO) it has not been addressed in any meaningful
manner.

To be sure, it will not be an easy task. But, especially if files that are
placed onto the network by companies trying to distribute demos and what-not
can be shared easily and securely, the network may be able to generate
revenue from those companies. It is imperative to have a stream of income
if the system is to remain viable and advancing. (Only kids can code for
free....and, even then, not for long).

I do have 2 oher small businesses to run, but I hope to find time to get a
BETA out by Dec 24th.
 
Perfectly on topic.

Thanks for the link!

Austin Ehlers said:
<snip>

http://suprnova.org

Contains just about any (non-adult) files that are out on the
internet. Careful, though, as the MPAA is definitely aware of the
site.

There are many others. http://btsites.tk/ has links to lots of BT
sites.

(Sorry if this is too OT, as I can't find the original message.)

Austin
 
To be sure, it will not be an easy task. But, especially if files that are
placed onto the network by companies trying to distribute demos and what-not
can be shared easily and securely, the network may be able to generate
revenue from those companies. It is imperative to have a stream of income
if the system is to remain viable and advancing. (Only kids can code for
free....and, even then, not for long).

Um, there's a *vast* amount of high quality open source software which
disproves your last point. Or do you believe that Apache was written by
kids?
 
Jon Skeet said:
Um, there's a *vast* amount of high quality open source software which
disproves your last point. Or do you believe that Apache was written by
kids?

I believe that Apache is an aberration. The vast majority of open source
projects don't survive for more than 4 years (most less than 2).

But, don't believe me, just go take a look around SourceForge and see for
yourself. I'd love to know the % of projects on SourceForge that are no
longer being updated or that have been abandoned.

There is one open source project I'd love to start myself.....but that'll
have to wait. I have to make a living right now.
 
Jim Hubbard said:
I believe that Apache is an aberration. The vast majority of open source
projects don't survive for more than 4 years (most less than 2).

But, don't believe me, just go take a look around SourceForge and see for
yourself. I'd love to know the % of projects on SourceForge that are no
longer being updated or that have been abandoned.

There is one open source project I'd love to start myself.....but that'll
have to wait. I have to make a living right now.

Yes, there are plenty of projects which have been abandoned. Sometimes
that's because better alternatives have been found. Sometimes that's
because they're basically finished, and work fine.

There are plenty which are on-going, however. Apache is just one of
them. BitTorrent is another, of course - that seems to be viable to me,
given the huge number of users...
 
Yes, there are plenty of projects which have been abandoned. Sometimes
that's because better alternatives have been found. Sometimes that's
because they're basically finished, and work fine.

There are plenty which are on-going, however. Apache is just one of
them. BitTorrent is another, of course - that seems to be viable to me,
given the huge number of users...

BitTorrent would also seem to be a natural for an MP3 site like iTunes.

Admittedly MP3s aren't that large, but BitTorrent would reduce server load
and bandwidth charges for the service and make downloading the most popular
MP3s almost instantaneous.

Preaching, eh? I used to do that. Good luck to you.

Jim Hubbard
 
Back
Top