Data Mining - Revision history

Aplstudent at 06:13, 23 April 2016

2016-04-23T06:13:13Z

← Older revision		Revision as of 06:13, 23 April 2016
Line 1:		Line 1:
			To see an example of taking a dataset and applying machine learning techniques go to https://github.com/aplstudent/Data_Anylasis/blob/master/example.R

	Intro		Intro

Aplstudent at 06:39, 14 April 2016

2016-04-14T06:39:34Z

← Older revision		Revision as of 06:39, 14 April 2016
Line 10:		Line 10:
	With the growth of the amount of data generated by the internet machine learning has undergone a huge explosion as the size of data has become huge. There are applications using the algorithms that we use everyday. Face recognition, voice recognition, word prediction, spam detection, cancer detection, genetics, hand written digit recognition, text analytics.		With the growth of the amount of data generated by the internet machine learning has undergone a huge explosion as the size of data has become huge. There are applications using the algorithms that we use everyday. Face recognition, voice recognition, word prediction, spam detection, cancer detection, genetics, hand written digit recognition, text analytics.

	There are three type of learning. Supervised, unsupervised and ~~reenforcment~~. Supervised learning tasks are ~~regrssopm amd~~ classification, reinforcement learning is for stuff like the stock market, unsupervised is ~~for other things~~.		There are three type of learning. Supervised, unsupervised and re-enforcment. Supervised learning tasks are regression and classification, reinforcement learning is for stuff like the stock market, unsupervised is a really advanced topic.

	Overview of the process		Overview of the process
Line 20:		Line 20:

	Collecting Data:		Collecting Data:
	There are many interesting data sets that are available for exploration Uci machine learning ~~repossitory~~, kaggle, reddit/r/datasets ckan(government documents) government docs, library of congress, website api’s,		There are many interesting data sets that are available for exploration Uci machine learning repository, kaggle, reddit/r/datasets ckan(government documents) government docs, library of congress, website api’s,

			clean the data:
			decide what to do with missing variables, explore how complete the data is. Create graphs and statistics that are relevant to the variables. Find the type of each variable and convert categorical to numeric.

			The model:
			There are lots of models out there that work with different degrees of success. There are linear models, clustering, networks, and trees.

			evaluation:
			create a separate set of data to use as cross validation in order to avoid overfitting the model to the data. There are many metrics, such as accuracy, recall, F1.It's best to choose one and stick with it.

	What to do when struggling:		What to do when struggling:

	R and python both have complete documentation so you can find information about any function in it. ~~ALso~~ if you press "tab" after a dot		R and python both have complete documentation so you can find information about any function in it. Also if you press "tab" after a dot
	ipython will show you the options you can write next. In Rstudio this is done automatically. There is also a very active community of		ipython will show you the options you can write next. In Rstudio this is done automatically. There is also a very active community of
	users and developers so information on everything is plentiful.		users and developers so information on everything is plentiful.
Line 31:		Line 40:

	Python Tutorials and blogs.		Python Tutorials and blogs.
	If you've never touched python start by doing the first 20 ~~exercies~~ of this:		If you've never touched python start by doing the first 20 exercises of this:
	[[http://learnpythonthehardway.org/book/ex1.html LPTHW]]		[[http://learnpythonthehardway.org/book/ex1.html LPTHW]]

Aplstudent at 06:30, 14 April 2016

2016-04-14T06:30:07Z

@@ Line 1: / Line 1: @@
-Every field is generating huge amounts of data. Data mining has been applied to the fields of Artificial Intelligence, Humanities, Biology, Physics, Marketing, Operations. The techniques of statistics, machine learning, programming, problem solving, information science, communication and visualization have combined to form the field of Data Science.
+Intro
-The goal of this project is to explore the data mining research field and acquire skills using python and R
+Data Science is a buzz word that means different things to different people. It can be the use of any amount of data and data analysis in a scientific process. It is also a growing field at the intersection of machine learning, statistics, information science and business.
-Current Research project:
+History as I know it
-Clustering and Data Visualization in the Chronicling America Archives.
+Machine learning is a sub field of artificial intelligence. It focuses on how to make algorithms that predict, classify outputs of a data set. This field utilizes the theory of discrete math, and graphs to provide a rigorous understanding and process for knowledge. Machine Learning itself is part of Data Mining, which is finding trends and patterns in large sets of data.
-teaching modules to be added:
+Applications
 What to do when struggling:
@@ Line 31: / Line 28: @@
 users and developers so information on everything is plentiful.

Aplstudent: Data Mining landing page

2016-03-20T01:59:02Z

Data Mining landing page

New page

Every field is generating huge amounts of data. Data mining has been applied to the fields of Artificial Intelligence, Humanities, Biology, Physics, Marketing, Operations. The techniques of statistics, machine learning, programming, problem solving, information science, communication and visualization have combined to form the field of Data Science.

The goal of this project is to explore the data mining research field and acquire skills using python and R
The tutorials I think are helpful will be linked here. There will be a series of tutorials on a variety of useful packages linked to github.

Current Research project:
Clustering and Data Visualization in the Chronicling America Archives.
One can use the api (follow my code)
Or get a builk download of files
I am working on getting access to the entire collection

teaching modules to be added:
using APL resources to use ipython 2 notebook
Rstudio on windows
-packages
setting up iep and Anaconda on windows
-packages and pip
python and R
loading various formats data
web scraping( html/xml/css)
the power of pandas
starting with sklearn
introduction to theano and
progression of models to make

What to do when struggling:

R and python both have complete documentation so you can find information about any function in it. ALso if you press "tab" after a dot
ipython will show you the options you can write next. In Rstudio this is done automatically. There is also a very active community of
users and developers so information on everything is plentiful.

Programming is a skill that becomes easier to learn the longer you do it. The process I follow to build understanding of the problem is as follows
start researching!
find an example of the function being used correctly by googling : "(name of package/function) and 'example')"
read a tutorial on how to use it: google: "(name of package/function) and 'tutorial')"
read the documentation for the package: google ("name of package/function" and "documentation")
use python help() and R ? to read documentation in environment.
start programming!
simplify your problem to make the smallest possible piece of code work
repeat research step if new problems come up.
build up the problem and keep checking code until finished.

Python Tutorials and blogs.
If you've never touched python start by doing the first 20 exercies of this:
[[http://learnpythonthehardway.org/book/ex1.html LPTHW]]

Then follow the scipy primer on this wiki

to practice python skills [[https://www.reddit.com/r/dailyprogrammer Subreddit]]

To learn machine learning:
This progression of tutorials take you from a little knowledge of numpy to a modern neural net
with all the bells and whistles. It'll make you feel amazing about what you can do with python.
first tutorial : [[https://iamtrask.github.io/2015/07/12/basic-python-network/ NeuralNet]]
All: [[http://iamtrask.github.io/]]

intro to working with datasets:
https://www.kaggle.com/c/titanic