How Artificial Neural Networks can be used for Data Mining

BUSINESS & MANAGEMENT, SYSTEMS & TECHNOLOGY   |   4 minutes  |   May 3, 2019

GetSmarter Blog Image GetSmarter Blog Image  

You’ve probably heard that data is the new gold, or the new oil. Regardless of which, both are true, as data is a valuable resource that takes effort to mine, but once extracted, makes up for the raw material used in creating other valuable products.1

As the big data trend continues to see businesses amass droves of data, the need for automated processing to handle and understand such volumes has become essential2 – enter neural networks. The ability of neural networks to detect and assimilate relationships between a large number of variables is becoming increasingly relevant to businesses that wish to effectively mine and understand big data.3

Understanding neural networks

The Artificial Neural Network (ANN) bases its assimilation of data on the way that the human brain processes information. The brain has billions of cells called neurons that process information in the form of electric signals. External information, or stimuli, is received, after which the brain processes it, and then produces an output.4

Similarly, ANN receives input through a large number of processors that operate in parallel and are arranged in tiers. The first tier receives the raw input data, which it then processes through nodes that are interconnected and have their own packages of knowledge and rules. The processor then passes it on to the next tier as output. Each successive tier of processors and nodes receives the output from the tier preceding it and further processes it; rather than having to process the raw data anew every time.5

Neural networks modify themselves as they learn from their robust initial training and then from ongoing self-learning that they experience by processing additional information.6 A simple learning model applied by neural networks is the process of weighting input streams in favour of those most likely to be accurate.7 That means a preference is put on the input streams that have a higher weight; and the higher the weight, the more influence that unit has on another. The process of reducing predictable errors through weight, is done through gradient descent algorithms.8 Finally, output units are the end part of the process; this is where the network responds to the data that was put in initially, and can now be processed.9

GetSmarter Blog Image GetSmarter Blog Image  

The Artificial Neural Network (ANN) bases its assimilation of data on the way that the human brain processes information. The brain has billions of cells called neurons that process information in the form of electric signals. External information, or stimuli, is received, after which the brain processes it, and then produces an output.

Neural networks in business

Most retail companies understand that their data provides decision-making information and is a valuable resource in business operations. As technology grows, businesses are leveraging neural networks for predictive analytics to fully harness the benefits of data streams.10

ANNs can learn and model non-linear and complex relationships, and have the capacity to manage the reality between the relationship of inputs and outputs, as this is seldom linear or simple. There is a tendency to generalise and attribute unseen relationships on unseen data. ANNs also don’t impose any restrictions on the input variables, unlike other prediction techniques.11

Here are some examples of how neural networks are applied across many industries and organisations to great effect.12

1. Forecasting

Most departments in a business lean heavily on forecasting data on a daily basis in order to manage their sales, stock, workforce and so forth. Typically, forecasting problems are complex and traditional forecasting models apply limitations to data in order to control these complex, non-linear relationships. Applied correctly, ANNs can effectively forecast without limitations on data because of its modelling ability to extract unseen features and define relationships.

2. Character and image recognition

ANNs are able to take in a multitude of inputs and process them in hidden and complex, non-linear relationships. This makes them perfectly positioned for character recognition, such as handwriting, which is being effectively used in fraud detection and even national security assessments. The following are all made possible through ANNs: Image recognition, such as facial recognition on social media platforms; cancer detection for healthcare industries; and satellite imagery processing for agricultural and military use.

3. Data mining

Neural networks are used for effective data mining in order to turn raw data into useful information. Neural networks look for patterns in large batches of data, allowing businesses to learn more about their customers which directs their marketing strategies, increase sales and lowers costs.13

What is data mining?

Data mining, also known as data discovery and knowledge discovery, is the process of analysing hidden patterns of data against criteria in order to categorise the data into useful information.14

This mined data can be used to inform marketing decisions, improve sales and cut down on costs.15 Here are the general steps taken when mining data:

1. Understanding the business

In order to create accurate data mining goals, it’s important to first understand your business’s goals and needs, and its current situation. Then a good data mining plan can be drafted.16

2. Understanding the data

The first steps for understanding data is to collect, load, and integrate the data from all available sources. Ensuring the data is clean, coherent and consolidated, is important for the data to be effectively mined.17

3. Preparing the data

This part of the data mining process requires a fair amount of time and attention. The data needs to be cleaned, constructed and formatted so it can be mined by software allowing for deeper data exploration to take place. First work with a sample segment of the data and iteratively apply various preparation steps to filter and cleanse to get the data, to ensure it’s in a useable format. Once preparation on the sample set is complete, start the actual steps for data mining on the entire data set, and review the results to confirm if the preparation method is sufficient.18

4. Selecting a model

The modelling techniques used, need to be chosen for the prepared data set. Test scenarios should be created to test the validity of the selected models, and should be carefully assessed by all stakeholders. The best test scenario for the business should be applied to the entire data set.19

5. Evaluating the data

New patterns may be brought to light through the data mining process, and this may result in the initial business objectives identified in step one being re-evaluated and revised. Gaining understanding into your business through data mining is a continuous process.20

6. Presenting the data

Business insights revealed by data mining process need to be presented or deployed in the business, in such a way that stakeholders can effectively make use of the information.21

GetSmarter Blog Image GetSmarter Blog Image  

Data mining, also known as data discovery and knowledge discovery, is the process of analysing hidden patterns of data against criteria in order to categorise the data into useful information.

How neural networks are being used for data mining

In Professor Arackioraj’s paper, “Applications of neural networks in data mining”, he notes that finding information that is hidden in data is as difficult as it is important.22 Neural networks mine data in areas such as bioinformatics, banking, and retail. Using neural networks, data warehousing organisations can harvest information from datasets to help users make more informed decisions through neural network’s ability to handle complex relationships, cross-pollination of data, and machine learning. Neural networks and AI technologies can carry out many business purposes with unstructured data, from tracking and documenting real-time communications, to finding new customers that automate follow-ups and flag warm leads.23

Until recently, decision-makers had to rely primarily on extracted data from structured, highly organised data sets, as these are easier to analyse. Unstructured data like emails and copy, are more difficult to analyse, and so have gone unutilised or simply ignored. Neural networks can now provide decision-makers with much deeper insight into the ‘why’ of a customer’s behaviour, which goes beyond what is provided in more structured data.24

In healthcare, an example of how neural networks are successfully mining data is shown by Imperial College London, where ANNs are used to produce optimal patient care recommendations for patients with sepsis.

GetSmarter Blog Image GetSmarter Blog Image  

LinkedIn uses neural networks as well as linear text classifiers to pick up spam or abusive content in its feeds.26 “We also use neural nets to help understand all kinds of content shared on LinkedIn,” says Deepak Agarwal, LinkedIn’s vice president of Artificial Intelligence, “ranging from news articles to jobs to online classes — so we can build better recommendation and search products for members and customers.”

DialogTech uses neural networks to sort inbound calls into various categories, as well as to sort the leads against a quality score. The neural network carry out these actions based on the call transcriptions and keywords used in the call.

ANNs are becoming a more trusted and useful tool for finding solutions within unstructured data, thanks to their ability to model nonlinear processes. Problems such as classification, clustering, regression, pattern recognition, dimension reduction, structured prediction, machine translation, and anomaly detection are now being solved, making it easier for decision-makers to know the correct way forward, and take longer, more confident strides towards their business’s future.

  • 1 (Oct, 2018). ‘Like mining gold, extracting data value takes effort’. Retrieved from Forbes.
  • 2 Rouse, M. (Nov, 2018). ‘What is big data?’. Retrieved from TechTarget.
  • 3 Mahanta, J. (Jul, 2017). ‘Introduction to neural networks, advantages and applications’. Retrieved from Towards Data Science.
  • 4 Mahanta, J. (Jul, 2017). ‘Introduction to neural networks, advantages and applications’. Retrieved from Towards Data Science.
  • 5 Rouse, M. (Jul, 2018). ‘Special report: artificial intelligence apps come of age’. Retrieved from TechTarget.
  • 6 Marr, B. (Sep, 2018). ‘What are Artificial Neural Networks – a simple explanation for absolutely anyone’. Retrieved from Forbes.
  • 7 Marr, B. (Sep, 2018). ‘What are Artificial Neural Networks – a simple explanation for absolutely anyone’. Retrieved from Forbes.
  • 8 Mahanta, J. (Apr, 2017). ‘Keep it simple: how to understand gradient descent algorithm’. Retrieved from KDNuggets.
  • 9 Marr, B. (Sep, 2018). ‘What are Artificial Neural Networks – a simple explanation for absolutely anyone’. Retrieved from Forbes.
  • 10 Freyer, J. (Mar, 2019). ‘How to optimize predictive analytics and Machine Learning for your retail brand’. Retrieved from Forbes.
  • 11 Mahanta, J. (Jul, 2017). ‘Introduction to neural networks, advantages and applications’. Retrieved from Towards Data Science.
  • 12 Mahanta, J. (Jul, 2017). ‘Introduction to neural networks, advantages and applications’. Retrieved from Towards Data Science.
  • 13 Kenton, W. (May, 2018). ‘Data mining’. Retrieved from Investopedia.
  • 14 (Nd). ‘Data mining’. Retrieved from Techopedia. Accessed on 04 Apr, 2019.
  • 15 Kenton, W. (May, 2018). ‘Data mining’. Retrieved from Investopedia.
  • 16 Peterson, R. (Oct, 2018). ‘Six essential steps in the data mining process’. Retrieved from Barn Raisers.
  • 17 Patidar, S. (Aug, 2018). ‘Data mining process: cross-industry standard process for data mining’. Retrieved from DZone.
  • 18 (Jun, 2018). ‘10 best practices in data preparation’. Retrieved from Import.io.
  • 19 Patidar, S. (Aug, 2018). ‘Data mining process: cross-industry standard process for data mining’. Retrieved from DZone.
  • 20 Peterson, R. (Oct, 2018). ‘Six essential steps in the data mining process’. Retrieved from Barn Raisers.
  • 21 Patidar, S. (Aug, 2018). ‘Data mining process: cross-industry standard process for data mining’. Retrieved from DZone.
  • 22 Arockiaraj, M. (May, 2013). ‘Applications of neural networks in data mining’. Retrieved from Research Inventory.
  • 23 Loten, A. (Apr, 2018). ‘AI tools find new customers for companies’. Retrieved from Wall Street Journal.
  • 24 Rogers, A. (Jan, 2019). ‘The 80% blind spot: are you ignoring unstructured organizational data?’ Retrieved from Forbes.
  • 25 Wighton, K. (Oct, 2018). ‘AI doctor could boost chance of survival for sepsis patients’. Retrieved from Imperial College London.
  • 26 Morphy, E. (May, 2018). ‘What is a neural network and how are businesses using them?’ Retrieved from CMSWire.