Big Data, Data Mining and Analytics: Uses, Benefits, Challenges and Risks

The last topic showed what big data is and how it becomes knowledge. This topic asks the question a finance director will ask first: what is it worth, and what could go wrong? A courier company that reroutes its drivers using live traffic and map data can save millions of kilometres of driving and a very large amount of fuel.

A bank that scores every transaction for fraud in a fraction of a second can stop a theft before it completes. A retailer that finds which products customers buy together can change its shelves and its offers. But the same data can leak, mislead, discriminate or simply cost more than it earns.

This topic explains the uses and benefits of big data, how data mining and data analytics work and the main mining techniques, the challenges and risks of applying them in an organisation, and how to appraise a data project in the same way as any other investment. It builds on the big data principles and pyramid topic.

The next topic starts the study of advanced costing techniques with activity based costing.

Uses and benefits of big data

Only a small percentage of the huge amount of data available is ever analysed. The importance of big data is not how much there is but how it is used. Analysis may lead to cost reductions, new products and smarter decisions, and it is changing how the world uses business information.

A large courier company, for example, used data from vehicle sensors and online maps to rework drivers’ pickups and drop-offs in real time, which cut tens of millions of miles from its routes and saved millions of gallons of fuel.

Benefits

• Better decisions. Analysis of large data sets gives insights that guide strategic choices, and predictive models forecast trends, customer behaviour and risks.
• Customer experience and personalisation. Continuous interaction data lets a business tailor offers, products and services, which raises satisfaction and loyalty. Marketing reaches the right audience more accurately.
• Efficiency and productivity. Bottlenecks and inefficient processes are found and removed, supply chains are forecast better, and workflows improve.
• Lower costs. Waste is cut and resources are used better. Predictive maintenance finds equipment failure before it happens, which reduces downtime, and budgets improve with better expense forecasts.
• Competitive advantage. Eliminating inefficient operations and acting faster than rivals.
• Risk identification. Potential risks and weaknesses are found earlier, so management can act before they cost money, for example in fraud detection and credit scoring.
• Innovation. Unmet needs and market gaps show up, and user feedback speeds product improvement. Structured and unstructured data together give richer insight.
• Public benefit. Healthcare diagnosis, disease tracking, urban planning, disaster response.

Data mining

Data mining is the process of sorting large, complex data, structured or unstructured, to find patterns, trends and relationships that help solve business problems and support informed decisions. It depends on good preparation: gathering, collecting and preparing the data, then mining and analysing it. The main techniques are:

• Association rules. Find items or events that occur together, for example products bought together.
• Classification. Place items into known groups using methods such as logistic regression, for example good or bad credit risk.
• Clustering. Group items that are similar without predefined groups, using methods such as k-means and hierarchical clustering, for example customer segments.
• Regression. Estimate a relationship between a result and one or more drivers, using linear and multiple regression.
• Sequence and path analysis. Find the order in which events happen, for example the steps customers take before they buy or leave.
• Neural networks. Algorithms that imitate the way the brain works, processing data through nodes, and used for complex patterns.
• Decision trees. Tree-like structures that show possible decisions and their outcomes.

Data analytics

Data analytics is the process of examining data to obtain meaningful insights and draw conclusions, which help organisations make informed decisions and improve performance. It replaces guesswork in marketing, product development, content creation and customer service. It allows targeted content to be tested and fine-tuned on real-time data, and it shows how campaigns are performing.

For a management accountant it extends the familiar tools of variance analysis, forecasting and customer profitability to far larger data sets.

Challenges and risks

Mining big data in an organisation brings challenges that should be planned for, and risks that should be managed.

Challenges

• Poor data quality. Inaccurate, incomplete or inconsistent data compromises the result: garbage in, garbage out.
• Volume and complexity. The sheer size of the data affects processing speed and the usefulness of results, and unstructured data from many sources is hard to clean.
• Privacy and security. Most such data is sensitive and must not be open to tampering or copying, and big data systems are an attractive target for cyberattacks.
• Scalability. Mining methods must handle very large data.
• Interpretability. Complex models are hard to interpret, so managers may not trust or be able to explain the result.
• Ethics. Collecting and using data raises ethical questions: data used to train models may not be representative, so models can reinforce bias, profiling can disadvantage groups, and it may not be clear how data is used in automated systems.
• Cost and skills. Storage, processing platforms and scarce data scientists are expensive, and integrating with existing systems is hard.
• Information overload. Too much data can swamp decision makers and lead to analysis paralysis, so that the useful signal is lost in noise.

Risks

• Privacy and data protection risks: fines, claims and loss of trust.
• Risks of unorganised data: data that is not catalogued cannot be found, trusted or controlled.
• Cost risk: mining and managing big data may cost more than the benefit.
• Storage and retention risk: keeping data too long breaches the law and enlarges the damage of a breach; keeping it too briefly loses its value.
• Incompetent procedures: wrong methods or wrong people give wrong answers that look convincing.
• Poor-quality data: flawed decisions based on flawed data.

Controls follow from the principles in the previous topic: data quality checks and cleansing, access control and encryption, a clear data governance policy, staff with the right skills, consent and transparency, review of models for bias, a human check on important automated decisions, and a cost-benefit appraisal before the project starts.

Data protection law and data analytics in Nigeria

• Section 24 principles. Analytics must keep to lawfulness, fairness and transparency, purpose limitation, data minimisation, accuracy, storage limitation, and integrity and confidentiality. Data collected for one purpose may not simply be reused for a new analytic purpose without a lawful basis.
• Consent (section 26). Where consent is the basis, the controller must be able to prove it, and silence is not consent.
• Impact assessment (section 28). A data protection impact assessment is required before processing that is likely to be high risk. The 2025 implementation directive lists large-scale processing, new technologies, profiling and scoring, and significant automated decisions among the triggers.
• Automated decisions (section 37). A person has the right not to be subject to a decision based solely on automated processing, including profiling, that has legal or similarly significant effects, for example refusing a loan, subject to stated exceptions. A human check is the usual control.
• Security and breach (sections 39 and 40), and cross-border transfers. Security measures are required, a risky breach must be reported to the Commission within 72 hours, and transfers abroad need adequate protection or safeguards.

Treat compliance as part of the cost of every analytics project, and as a risk to be weighed against its benefits. Confirm current Commission notices before advising a real client.

Sources and references

• Nigeria Data Protection Act 2023, sections 24, 26, 28, 37, 39 and 40, and Part VIII (cross-border transfers).
• Nigeria Data Protection Commission, General Application and Implementation Directive 2025, in force from 19 September 2025.

These points are stated as at 6 November 2026.

Appraising a data project and reading a mining result

Use the cost-benefit layouts from the first topic, and the association rule measures below.

1. Net annual benefit. Annual benefits − running costs. Payback = initial cost ÷ net annual benefit.
2. NPV. Net annual benefit × annuity factor − initial cost. Break-even net benefit = initial cost ÷ annuity factor. Margin for error = (net benefit − break-even net benefit) ÷ net benefit.
3. Support of A and B. Transactions containing both ÷ all transactions.
4. Confidence of A to B. Transactions containing both ÷ transactions containing A.
5. Lift. Confidence ÷ the share of all transactions that contain B. A lift above 1 means that A and B occur together more often than chance.

Annuity factors are shown to four decimal places, money is in ₦ million to two decimal places unless stated, percentages to one decimal place. Each answer is rounded once, at the end.

Worked examples

1. Fuel saving.
   6% × 180.0 = ₦10.8m.
   Total benefits. 10.8 + 14.5 + 9.2 + 6.5 = ₦41.0m.
2. Net annual benefit.
   41.0 − 12.0 = ₦29.0m.
3. Annuity factor.
   Four years at 16% = 2.7982.
   Present value 29.0 × 2.7982 = ₦81.15m.
4. NPV.
   81.15 − 55.00 = +₦26.15m.
5. Payback.
   55.0 ÷ 29.0 = 1.90 years.
6. Break-even net benefit.
   55.0 ÷ 2.7982 = ₦19.66m a year.
7. Margin for error.
   (29.0 − 19.66) ÷ 29.0 = 32.2%.
8. Advice. The project is worth doing and has a wide margin. The benefits come from four different sources, so the company should check each: the marketing and fraud figures are the least certain. It must also add the cost of data protection compliance and the risk of poor data to the plan.

1. Support.
   600 ÷ 5,000 = 12.0% of all transactions contain both.
2. Confidence.
   600 ÷ 1,200 = 50.0% of bread buyers also buy milk.
3. Share of transactions with milk.
   1,000 ÷ 5,000 = 20.0%.
4. Lift.
   50.0% ÷ 20.0% = 2.5.
5. Meaning. Bread buyers are two and a half times as likely to buy milk as a typical customer, so the two are strongly linked. The store might place them near each other, offer a joint discount, or make sure that both are in stock together. Association does not prove cause, so the idea should be tested.

Expect variants: several years of changing benefits, a residual value, rules with three items, or a question that gives the support and asks for the number of transactions.

A lean information system

A lean information system supports lean manufacturing and is built on three principles. Elimination of waste: errors in data are waste, because wrong information leads to wrong decisions and correcting errors adds no value. Efficient flow: information should be available when it is needed, not before it is needed and not late; it is pulled by the user, not pushed out by the accountant.

Clarity: information must be presented in a form the user understands, because unused information has no value.

Why a traditional system fails. Frequent data-capture errors; monthly variance reports issued two weeks after month end (late, and pushed to managers instead of being available on request); absorption costing, which rewards producing for stock because overhead is spread over more units and stock absorbs overhead; and efficiency and capacity variances, which push managers to maximise output and so build stock, the opposite of lean aims.

Changes needed. Find and remove the causes of errors (manual forms, poor training; automate capture); cut the delay between event and record; give managers on-line access to cost data; drop absorption costing and value stock at direct material cost (activity-based management may still be used to monitor overhead); keep price and usage variances for materials but use non-financial measures for labour and flow, such as cycle time, throughput time, the ratio of idle to active time and, with just-in-time purchasing, order-to-delivery time.

Common mistakes

• Listing benefits without risks. A discussion needs both, and a conclusion.
• Confusing data mining with data analytics. Mining finds patterns in large data sets; analytics is the wider process of drawing conclusions for decisions.
• Naming techniques without a use. Give an example for each: association for products bought together, clustering for customer segments.
• Treating a pattern as a cause. Association and correlation do not prove cause.
• Forgetting cost. Platform, skills, storage and compliance can outweigh the benefit.
• Ignoring bias and privacy. These are central to the ethics and the law.

Quick revision

• Value lies in use: cost reduction, new products, smarter decisions.
• Benefits: decisions, personalisation, efficiency, lower costs, competitive advantage, risk identification, innovation.
• Mining techniques: association rules, classification, clustering, regression, sequence and path analysis, neural networks, decision trees. Analytics: examining data for conclusions and better decisions.
• Challenges: quality, volume, privacy and security, scalability, interpretability, ethics, cost and skills, overload. Risks: privacy, unorganised data, cost, storage and retention, incompetence, poor data.
• Nigeria Data Protection Act 2023: section 24 principles, section 26 consent, section 28 impact assessment, section 37 automated decisions, section 40 breach notice within 72 hours.
• Support = both ÷ all; confidence = both ÷ A; lift = confidence ÷ share with B.

How the marks are earned

• Discuss uses and benefits: point, explanation, business example; about one mark per developed point.
• Explain techniques: a definition and a use for each.
• Challenges and risks: separate the two, link each to a control, and conclude.
• Calculate: layout, figures and a conclusion.

Practice questions

These are original questions, written in the style of the Skills examination. Work each one before you open the answer.

Recall check: multiple choice

Section B style: written questions

Model answers

Spotted a mistake? Tell us and we will check it.