serialize vs database

  • Thread starter Thread starter mp
  • Start date Start date
M

mp

Hi all,
My current project is evolving over time as are my self inflicted
requirements.<g>
I'm now seeing the need to store info over time for comparisons

The project gathers daily info on stocks from web.
Up to 85 criteria can be chosen for what to look at.
at this point I am only looking at 10 or so but just starting
so that could evolve with time.
if i get data for 10 or so stocks that's between 100-850 'data points'
daily, and i need to watch trends over time so even at 100
that could be from 36500 to 365000 in a year(approx)
(even if i dont' want to see all 85 criteria today, i may find in the
future
that i wished i'd collected past data)

i found serializing one object to be easy.
i'm not so confident about being able to learn database.

I have a feeling however that xml files will become quickly unweildy
as compared a single .db file.

and obviously data mining/relationship comparisons would be much richer with
sql compared to comparing thousands of xml files.

Any thoughts? or simple tutorials on database in .net?
obviously i can find 100000 hits on google, but someone here may
already have culled some good ones :-)
thanks
mark
 
mp said:
i found serializing one object to be easy.
i'm not so confident about being able to learn database.

I have a feeling however that xml files will become quickly unweildy
as compared a single .db file.

and obviously data mining/relationship comparisons would be much richer with
sql compared to comparing thousands of xml files.

Any thoughts? or simple tutorials on database in .net?
obviously i can find 100000 hits on google, but someone here may
already have culled some good ones :-)
Click to expand...

Your hunch that a database might be the way to go is, I believe, correct.

Have you thought of buying a book?

For example:
http://www.amazon.com/Beginning-C-Databases-Novice-Professional/dp/1590594339/
 
My current project is evolving over time as are my self inflicted
requirements.<g>
I'm now seeing the need to store info over time for comparisons

The project gathers daily info on stocks from web.
Up to 85 criteria can be chosen for what to look at.
at this point I am only looking at 10 or so but just starting
so that could evolve with time.
if i get data for 10 or so stocks that's between 100-850 'data points'
daily, and i need to watch trends over time so even at 100
that could be from 36500 to 365000 in a year(approx)
(even if i dont' want to see all 85 criteria today, i may find in the
future
that i wished i'd collected past data)

i found serializing one object to be easy.
i'm not so confident about being able to learn database.
Click to expand...

Database is not so hard.
I have a feeling however that xml files will become quickly unweildy
as compared a single .db file.

and obviously data mining/relationship comparisons would be much richer with
sql compared to comparing thousands of xml files.
Click to expand...

Correct.

It is not easy/fast to update part of a single big XML file.

It is not easy/fast to search in many small XML files.

Database is the way to go.
Any thoughts? or simple tutorials on database in .net?
obviously i can find 100000 hits on google, but someone here may
already have culled some good ones :-)
Click to expand...

Order of learning:
SQL
ADO.NET - SqlCommand and SqlDataReader (for SQLServer, similar for
other databases)
ORM framework - LINQ to EF

As you have found out then there are plenty of stuff
available on the net - I don't have anything particular
tutorials to recommend.

Arne
 
Hi all,
My current project is evolving over time as are my self inflicted
requirements.<g>
I'm now seeing the need to store info over time for comparisons

The project gathers daily info on stocks from web.
Up to 85 criteria can be chosen for what to look at.
at this point I am only looking at 10 or so but just starting
so that could evolve with time.
if i get data for 10 or so stocks that's between 100-850 'data points'
daily, and i need to watch trends over time so even at 100
that could be from 36500 to 365000 in a year(approx)
(even if i dont' want to see all 85 criteria today, i may find in the
future
that i wished i'd collected past data)
Click to expand...
I see where you're going and my advise is don't overwhelm yourself and
your project. For the moment concentrate on gathering and storing the
transactional data. There are other tools which can be used to extract
business intelligence (tracking trends, historical comparisons etc.)
from the transactional data.
i found serializing one object to be easy.
i'm not so confident about being able to learn database.
Click to expand...
There is a lot to learn but you don't have to learn everything all at
once.
I have a feeling however that xml files will become quickly unweildy
as compared a single .db file.
yep
and obviously data mining/relationship comparisons would be much richer with
sql compared to comparing thousands of xml files.
yep
Any thoughts? or simple tutorials on database in .net?
Click to expand...

These links may or may not be useful to you but
http://msdn.microsoft.com/
is always a good place to start.

http://code.msdn.microsoft.com/SQLTutorials
http://msdn.microsoft.com/en-us/library/bb655891(v=VS.90).aspx

Don't worry about extracting business intelligence from your data just
yet. Once you have an understanding of relational databases and are
comfortable using them, then you can learn about MS SQL Integration
Services, Analysis Services and Reporting Services. These are powerful
tools and the way you're thinking, you'll really appreciate them.

regards
A.G.
 
Registered User said:
I see where you're going and my advise is don't overwhelm yourself and
your project. For the moment concentrate on gathering and storing the
transactional data. There are other tools which can be used to extract
business intelligence (tracking trends, historical comparisons etc.)
from the transactional data.

There is a lot to learn but you don't have to learn everything all at
once.


These links may or may not be useful to you but
http://msdn.microsoft.com/
is always a good place to start.

http://code.msdn.microsoft.com/SQLTutorials
http://msdn.microsoft.com/en-us/library/bb655891(v=VS.90).aspx

Don't worry about extracting business intelligence from your data just
yet. Once you have an understanding of relational databases and are
comfortable using them, then you can learn about MS SQL Integration
Services, Analysis Services and Reporting Services. These are powerful
tools and the way you're thinking, you'll really appreciate them.

regards
A.G.
Click to expand...

thanks for the input
I'll check those links out
mark
 
Jeff Johnson said:
Let's step back. Do you have any SQL experience at all? Did you ever use
ADO in VB?
Click to expand...

just a teensy bit, tried sqlite and olaf helped a lot
did a tiny bit with ado and iirc olaf had some helper classes
that hit the sqlite engine
i have a base concept of normalizing
i think i can figure out which tables to break into
primary and foreign keys etc
i think i can learn how to do in ado.net
but it's just another whole learning process that will take
another month or two from the actual project...
so i'm hoping to find some close examples i can tweak
for my use...that's usually the way i learn fastest...
not fast <g> ... guess i should say least slowest
;-)
i downloaded sqlserver expressa year ago or so and it was
way over my ability ...
i can probably look back at my vb6 db samples and get
some handle on the sqlite db option.
i think i need to dl a dotnet sqlite lib or something
mark
 
just a teensy bit, tried sqlite and olaf helped a lot
did a tiny bit with ado and iirc olaf had some helper classes
that hit the sqlite engine
i have a base concept of normalizing
i think i can figure out which tables to break into
primary and foreign keys etc
i think i can learn how to do in ado.net
but it's just another whole learning process that will take
another month or two from the actual project...
so i'm hoping to find some close examples i can tweak
for my use...that's usually the way i learn fastest...
not fast<g> ... guess i should say least slowest
;-)
i downloaded sqlserver expressa year ago or so and it was
way over my ability ...
i can probably look back at my vb6 db samples and get
some handle on the sqlite db option.
i think i need to dl a dotnet sqlite lib or something
Click to expand...

SQLite support does not come with MS .NET, so you will
need to download the ADO.NET provider for SQLite.

It really should not be that difficult to learn.

I think I have two simple SQLite examples on the
shelf. Ping me if interested.

Arne
 
( http://www.shahriarnk.com/Shahriar-N-K-Research-Embedding-SQL-in-C-Sharp-Java.html )

At the above link, you will find a very useful tutorial on developing database applications using C# as well as Java.

It describes in detail how to connect to the database, how to pass parametrized queries, call stored procedures and much more.

Do let us know if you find it useful.

Submitted via EggHeadCafe
ASP.NET In-Memory Image Control with Built-In Resizing of Posted File
http://www.eggheadcafe.com/tutorial...rol-with-builtin-resizing-of-posted-file.aspx
 
Arne Vajhøj said:
[]


I think I have two simple SQLite examples on the
shelf. Ping me if interested.

Arne
Click to expand...

examples are always appreciated!
I found one class on the web and have been looking at it

here's what i used to do in vb6
i'd created a field object with some properties
then sent a collection of those to this function to create the table
would i do somewhat similar in dotnet?

Public Sub CreateTable(TableName As String, FieldObjectsCollection As
Collection)
On Error GoTo CreateTable_Error
Dim ADOXtable As Table
Dim oField As cField
Set ADOXtable = New ADOX.Table
ADOXtable.Name = TableName
Set ADOXtable.ParentCatalog = moCatalog

With ADOXtable
'Add Columns to the table
With .Columns
For Each oField In FieldObjectsCollection
.Append oField.Name, oField.DataType
Next oField
End With 'columns

For Each oField In FieldObjectsCollection
If oField.AutoIncrement Then
.Columns(oField.Name).Properties("AutoIncrement") = True
End If
If oField.IsNullable Then
.Columns(oField.Name).Properties("Nullable") = True
End If
Next oField
End With 'table

'append tables to database`
moCatalog.Tables.Append ADOXtable

'logEntry "Check primary key "
For Each oField In FieldObjectsCollection
If oField.IsPrimary Then
Dim oKey As ADOX.Key
Set oKey = New ADOX.Key
With oKey
.Name = "PrimaryKey"
.Type = adKeyPrimary
.RelatedTable = TableName
.Columns.Append oField.Name
End With
ADOXtable.Keys.Append oKey
Exit For 'just one primary key
End If
Next oField

'Clean up
Set ADOXtable = Nothing
msDbTableName = TableName

ExitHere:
On Error GoTo 0
Exit Sub

CreateTable_Error:
LogError ProcName
End Sub
 
examples are always appreciated!
I found one class on the web and have been looking at it
Click to expand...

See below.
here's what i used to do in vb6
i'd created a field object with some properties
then sent a collection of those to this function to create the table
would i do somewhat similar in dotnet?
Click to expand...

Probably, but it is not how it is typical done in .NET.

Suggested order for learning:
1) Connection, Command and DataReader
2) DataAdapter and DataSet
3) Entity Framework

Arne

=====

using System;
using System.Data;

using System.Data.SQLite;

namespace E
{
public class TestClass
{
public static void Main(string[] args)
{
SQLiteConnection con = new SQLiteConnection(@"Data
Source=C:\TestEmb.db;Version=3;Password=hemmeligt;New=True");
con.Open();
SQLiteCommand cre = new SQLiteCommand("CREATE TABLE tbl (id
INTEGER NOT NULL, txt NVARCHAR(50), PRIMARY KEY(id))", con);
cre.ExecuteNonQuery();
for(int i = 0; i < 3; i++)
{
SQLiteCommand ins = new SQLiteCommand("INSERT INTO tbl
VALUES(" + i + ",'Test" + i + "')", con);
ins.ExecuteNonQuery();
}
con.Close();
}
}
}

using System;
using System.Data;
using System.Data.SQLite;

namespace E
{
public class TestClass
{
public static void Main(string[] args)
{
SQLiteConnection con = new SQLiteConnection(@"Data
Source=C:\TestEmb.db;Version=3;Password=hemmeligt");
con.Open();
SQLiteCommand sel = new SQLiteCommand("SELECT * FROM tbl",
con);
SQLiteDataReader rdr = sel.ExecuteReader();
while(rdr.Read())
{
Console.WriteLine((long)rdr[0] + " " + (string)rdr[1]);
}
rdr.Close();
con.Close();
}
}
}
 
Registered User said:
[] ?

These links may or may not be useful to you but
http://msdn.microsoft.com/
is always a good place to start.

http://code.msdn.microsoft.com/SQLTutorials
http://msdn.microsoft.com/en-us/library/bb655891(v=VS.90).aspx

Don't worry about extracting business intelligence from your data just
yet. Once you have an understanding of relational databases and are
comfortable using them, then you can learn about MS SQL Integration
Services, Analysis Services and Reporting Services. These are powerful
tools and the way you're thinking, you'll really appreciate them.

regards
A.G.
Click to expand...

been following the links, it's taken me to an example to download
http://www.4shared.com/file/TnR6qUzE/data.html
when i go there i get to 4shared site which says 4shared extension not
installed
is this a safe site and exe as far as you know?
thanks
mark
 
Arne Vajhøj said:
examples are always appreciated!
I found one class on the web and have been looking at it
Click to expand...

See below.
here's what i used to do in vb6
i'd created a field object with some properties
then sent a collection of those to this function to create the table
would i do somewhat similar in dotnet?
Click to expand...

Probably, but it is not how it is typical done in .NET.

Suggested order for learning:
1) Connection, Command and DataReader
2) DataAdapter and DataSet
3) Entity Framework

Arne

=====

using System;
using System.Data;

using System.Data.SQLite;

namespace E
{
public class TestClass
{
public static void Main(string[] args)
{
SQLiteConnection con = new SQLiteConnection(@"Data
Source=C:\TestEmb.db;Version=3;Password=hemmeligt;New=True");
con.Open();
SQLiteCommand cre = new SQLiteCommand("CREATE TABLE tbl (id
INTEGER NOT NULL, txt NVARCHAR(50), PRIMARY KEY(id))", con);
cre.ExecuteNonQuery();
for(int i = 0; i < 3; i++)
{
SQLiteCommand ins = new SQLiteCommand("INSERT INTO tbl
VALUES(" + i + ",'Test" + i + "')", con);
ins.ExecuteNonQuery();
}
con.Close();
}
}
}

using System;
using System.Data;
using System.Data.SQLite;

namespace E
{
public class TestClass
{
public static void Main(string[] args)
{
SQLiteConnection con = new SQLiteConnection(@"Data
Source=C:\TestEmb.db;Version=3;Password=hemmeligt");
con.Open();
SQLiteCommand sel = new SQLiteCommand("SELECT * FROM tbl",
con);
SQLiteDataReader rdr = sel.ExecuteReader();
while(rdr.Read())
{
Console.WriteLine((long)rdr[0] + " " + (string)rdr[1]);
}
rdr.Close();
con.Close();
}
}
}
Click to expand...

thanks Arne,
will commence 'playing'
:-)
mark
 
[]
cool i ran your sample
so i can just expand this command and add my 85(or however many)
fields?
new SQLiteCommand("CREATE TABLE tbl (id INTEGER NOT NULL, txt NVARCHAR(50),
PRIMARY KEY(id))"

(that's assuming for now i put them all in one table)

seems like it will become one big line, but i guess thats okay?

thanks

mark
 
[]
cool i ran your sample
so i can just expand this command and add my 85(or however many)
fields?
new SQLiteCommand("CREATE TABLE tbl (id INTEGER NOT NULL, txt NVARCHAR(50),
PRIMARY KEY(id))"

(that's assuming for now i put them all in one table)

seems like it will become one big line, but i guess thats okay?
Click to expand...

That should be OK.

I think you can get GUI tools for creating tables for
SQLite so you don't need to do it programmtically.

Arne
 
Arne Vajhøj said:
Arne Vajhøj said:
On 26-01-2011 15:38, mp wrote:
I think I have two simple SQLite examples on the
shelf. Ping me if interested.

examples are always appreciated!
Click to expand...
[]
cool i ran your sample
so i can just expand this command and add my 85(or however many)
fields?
new SQLiteCommand("CREATE TABLE tbl (id INTEGER NOT NULL, txt
NVARCHAR(50),
PRIMARY KEY(id))"

(that's assuming for now i put them all in one table)

seems like it will become one big line, but i guess thats okay?
Click to expand...

That should be OK.

I think you can get GUI tools for creating tables for
SQLite so you don't need to do it programmtically.

Arne
Click to expand...

i may look into that when i get time, but programmatcaly might even be
faster than gui. the other day when i decided to make the class with the 85
properties, i wrote a quick app to write the class .cs file by reading my
list of 85 items and turning each one into a private variable name and
matching public property with corresponding gets and sets...faster than
typing all that in by hand, let alone doing from a gui.
:-)
 
the other day when i decided to make the class with the 85 properties, i
wrote a quick app to write the class .cs file by reading my list of 85
items and turning each one into a private variable name and matching
public property with corresponding gets and sets...faster than typing all
that in by hand, let alone doing from a gui.
Click to expand...

Ah, you've discovered the joy of code generation!
 
Peter Duniho said:
[]
Ah, you've discovered the joy of code generation!
Click to expand...

And soon he will discover that for the most part, code that is generated
by other code should be used only by other code. :) At least, not when
the code-generation input doesn't require explicitly typing in all 85
members by hand _somewhere_ (e.g. .xsd schema turned into a container
type).

Ironically, I suspect that in this particular situation, there never
really should have been a type with 85 properties.
Click to expand...

i instinctively knew i was doing something wrong!
the dictionary is obviously far superior

Instead, it probably
would have made more sense to use a dictionary/indexer/etc. and have the
caller just provide the name as a string, rather than having it baked into
the code.
[]


Live and learn. :)

Pete
Click to expand...

i'm trying to, and you guys are making it much less
painful than it would be if i didn't have this miraculous
ng source of assistance.
mark
 
( http://www.shahriarnk.com/Shahriar-N-K-Research-Embedding-SQL-in-C-Sharp-Java.html )

At the above link, you will find a very useful tutorial on developing database applications using C# as well as Java.

It describes in detail how to connect to the database, how to pass parametrized queries, call stored procedures and much more.

Do let us know if you find it useful.
Click to expand...

It gives an OK overview of .NET data access in 2004 and Java
data access in 1998.

Unfortunately the calendar says 2011.

Arne
 
Back
Top