What constitutes 'Testing'? Is simply running the code during development 'Testing'?

A test is an experiment. You have a hypothesis, which is normally governed by a specification, such as "When I enter a username and password that I know to be valid and click login, I am brought past the login screen and to the dashboard." or "When I log in to the program with 10,000 simultaneous users, performance is no less than 95% of that when there are 1,000 simultaneous users".

All tests start with this hypothesis. The scope of the hypothesis may be limited to a small portion of the program, like a single function or component. It may involve multiple components. It may contain known good data, known bad data, or random data. There's a lot of different things you can shove into this hypothesis. But whatever it is, the hypothesis comes from a part of the product specification.

The next part of the test is the experiment itself. You perform the action and collect the result. This might be a Selenium or JUnit test. It might be a human clicking buttons. It might be a batch job that does routine checks. But someone has to perform the action and collect observations.

Then the observation is compared to the original hypothesis to determine if it passes. If you can't tell whether or not it passes, then you need to refine the criteria. This is why things like "fun" are so hard to test for.

So if you don't have a formal hypothesis and a formal comparison of observations to that hypothesis, then sorry, in my books you're not doing testing.

As a former developer, I can tell you a good rule of thumb:

If you're trying to make it work, you're developing.
If you're trying to make it break, you're testing.

That is really up to your organization and your relationship with your stakeholders.

In most cases, testing is just a label with no specific meaning. The ultimate goal is to deliver a high-quality product. "Testing" is just a label for a kind of activity that might help you reach that goal. You can define that label however you want, but it is your responsibility to deliver something that meets your quality goals.

That said, your arrangement with your stakeholders (the people who will interact with your software) may stipulate how and/or when testing should take place. If you have that kind of arrangement, you will need to use those stipulations to decide whether running software on your desktop meets their definition of testing.

TLDR; The specific terminology that probably describes what he is asking for is smoke testing.

The long version: You are arguing with your manger instead of working towards a solution. Terminology is important, but so is communication and getting the job done.

You haven't done unit or integration testing, you probably haven't done any performance or usability testing, there are all kinds of specific test that you haven't done -- forget about all of them.

Focus on what is being asked, and not the terminology used. Your manager wants you to document at least some of what you have verified about what you have built. What have you verified that can usefully be reported?

Write something up that gives him what he needs, if there is a specific terminology for what he is asking for, introduce him to it, if there's not, and you are uncomfortable using the word "tested", then use another word that means what he wants. Call it verified, examined, checked, inspected, scrutinized or Fred. Whatever works and makes it clear that you didn't just bang on the keyboard until your hands got tired and then call that done.

Finally, get a clear statement of what "done" means, both for the overall project and for the documentation that he wants from you.

Testing is validating a situation on a set of conditions. In your case you validate if it runs, you could say "I am going to test if it builds". This would mean you are actual testing something.

From a SDLC perspective you are actual just coding and checking if you think it works and is good enough, this it not testing. Testing is actually a bit more structured and hopefully a repeatable way of validating the same. Often it is also a different phase in the life-cycle, unless it is test-driven development or an other automated testing effort, which would be exercised parallel with developing code. Test (case) design could be done in parallel, but not by one person.

In your-case if management wants to really minimize on testing efforts describe which steps have to be taking to release a feature to a production environment in a definition of done. This definition should include that it builds and that the happy path of feature is executed to verify it is complete. I guess some developers could build systems without ever running it, but the time of punch-cards is far behind us. You could skip testing efforts like automated, exploratory and or usability testing.

Agree on a definition of done! Minimize it for now and extend it in the future when you run into trouble.

I get the feeling you are prototyping as you say the following in a comment:

the point of the project is to develop something and see if it's a viable product

Be aware that because you are taking shortcuts (e.g. skipping testing) you are creating technical debt. If the product is viable it might need a clean and well tested reimplementation, because taking shortcuts now will mostly lead to a product that is unmaintainable in the long run. Discuss technical debt with your stakeholders so they have a honest view on the future. Selling your minimal testing efforts as testing by placing them in a sheet with a timestamp sounds as lying to yourself and to your stakeholders and clients. Take technical debt as a serious issue. Be warned! :)

Without testing, how would you know it works right? And if it does not have to work right, why would you spend resources on it? When it passed tests, it means is is "DONE" - and you need to agree what "DONE" means.

Testing is the difference between Wally (from Dilbert cartoon) saying "it worked for me, my test file in my browser did not crashed" and saying "we can expect it will produce correct results for any input by any of our customers".

But the "testing" you did was developing. Code might work for other people processing different input files, or might not. Testing (by other people using different inputs and possibly different workflow) will determine if code is "done".

Multiple Levels of Done - you need to find out what is your boss' definition of done.

Good testing costs resources and time. If you can release "beta version", your customers will test it for you, give you input files which will break your assumption on input, etc (for no cost to your boss), but there is price: your customers will see failures of your code. It is business decision if it makes business sense to spend resources to present better products to your customers. I.e. first version of Windows which was usable was 3.1, previous versions were junk for beta testers.

It could be a perfectly valid approach to release beta version (without spending too much resources on it), so customers can see the product and try it in "beta" - if they have no expectations for it to work, produce valid input etc ("release early, release often"). It allows you to face customer's expectations before lots of resources were spent on something what customers don't care about. Talk to your boss, we have no idea what s/he wants.

As a side note, I would never argue with a boss for two hours like that. Instead, I would ask for time-out to research the differences. It might be that you have different definitions of "done", and your boss' definition might be perfectly fine for the product you work on. Getting some training in "agile development" would be beneficial for you - google is your friend.

The TL;DR answer: No.

The explanation:

Running code during development, on the developer's machine, is simply iterative development.

Running it on another system (one that does not have the development environment and does not compile the code first) can be considered smoke-testing.

That said, here's what I'd suggest you do, given that your application is a proof-of-concept trial (and my experience with these is that a distressing number of them become production code far too quickly):

• Before you do anything else, ask your manager what level of testing he expects. You want to make sure that both of you are using the same definitions or you'll be talking past each other. At this point, it matters less that the definitions are correct than that you are both communicating (which is a bad position to be in, but that's life).
• You should also ask your manager why there has been a change from the original "no need to test because this is a proof of concept". That will give you a better idea of why your manager is insisting you document some form of testing.
• If (as I suspect) there is a requirement from further up the food chain that proof of testing be shown, I'd recommend that your proof of testing be something like the following:
  • List installation version (i.e. you're running the application on a different machine) and the actions performed. Call this a smoke test.
  • For each listing in your spreadsheet, enter the modules you covered. So, for instance, you logged in and added a user, then logged out and logged in as that user. Your smoke tests covered login and user management.
  • Have a separate listing of all modules in the application and all functions within each module. Yes, it's work, but I suspect you're going to need a certain amount of covering your anatomy here.
  • Include a list of what you have not tested, and why (the "why" in your case is going to be "no time"). This list would cover the functions you expect to receive little use, the error handling (which is out of scope for smoke tests), exotic inputs, and so forth. If you're using anything resembling an agile process, make sure to be explicit that your tests cover only the happy path or steel thread - the minimum set of functionality that makes the application viable with correct inputs
  • Also include in your documentation that you're making no guarantees about what will happen if the application is used in any way outside your happy path testing. In my experience caveats like this get ignored, but they're useful to point to when the blamestorming starts.
• Finally, good luck. What I'm seeing in your question suggests that the politics in your workplace aren't the best. That's never fun to deal with.

Your problem is very simply a confusion of semantics. The word "testing" is too broad, hence allowing wildly different interpretations. It just makes no sense to talk about "testing" without saying what you actually mean.

Someone mentioned "definition of done". This is what you are looking for; it is a contract between you and your customer. The DoD can be as fuzzy or specific as you (and your customer, even if it is internal to your company) like. Part of the DoD is a metric of when the software is considered to be "correct". As long as you and your customer both agree to it, and your code fulfills it, you are in the clear.

In your scenario, the metric of your DoD could be: "the compiled program can be started manually with a trivial input on the developers laptop". For a DoD, that is perfectly acceptable. Everybody will know that the software will likely fail immediately when confronted with the real world, but for a prototype/proof of concept that may just be fine.

the question is: does running code equal testing?

if it means launching the code and to observe it runs and loads the page or initial screen, yes, it would be considered a test - a smoke test if you will.

but there's more to testing than just a smoke test. as others have mentioned, you would do justice to the project by performing other types of test e.g. functional test or even non-functional tests. There's UI or API testing. Testing against the sw requirements or user stories would be a good place.

from what I gather, the project needs at least a Test Lead, snr test analyst or better yet a Test Manager! Then you will have a test process, defect management, test design, test analysis, test strategy & test closure.

Developing without doing unit testing is, in my experience, similar to taking a used car for a test ride, without taking a look at the engine or any other components. Yes, you might make a long ride and experience no issues, but are you confident enough that this car is worth paying for?

If this is how you do your development, it's entirely possible that iteratively running code during your development cycle might be enough to cover a great portion of issues, without writing tests explicitly, for a smaller project. However, in my experience, doing this alone is not enough to ensure the lowest amount of production issues you can ensure as a developer.

So, again, in my personal experience:

1. Not doing unit testing while coding (not necessarily TDD, but just testing tiny chunks of code while you code) has a couple of issues.

   By "running and seeing what happens" you observe the system as a "black box", evaluating its quality through the UI (or whatever results you get from the program).

   Perhaps you write a chunk of code, then try to build, and repeat until you get rid of all the syntax errors. Then you try some inputs and it looks promising. But then you try some other, and it doesn't work - it's time to debug!

   Depending on your skills, this might be fun, it's like solving puzzles throughout the day. So, this tiny bug you found in a certain function just happened to manifest itself under right conditions, but the underlying problem is that this tiny function doesn't do what it's supposed to do.

   Unit testing tries to catch these errors in functions while you write them, even if those errors would be very hard to provoke in very rare conditions, and that's what adds confidence in your code to your manager and stakeholders.

   It's also efficient - it's very easy to prove a function is doing what it's supposed to do using a couple of quick tests. It also promotes writing code which is easily testable, which is generally as purely functional as possible, with as little state as possible. These functions are easy to test, and you can rest assured they will work in all conditions.

2. What you are doing while developing is a sort of a manual integration testing. But not doing automated unit + integration testing means you are risking regression bugs in your commits, and making it harder to refactor your code or safely introduce changes.

   If your CI server is running a bunch of tests on each commit, and they are all green, it adds confidence in your code.

   Refactoring the code is much easier too. Needing to rewrite a part of an app is not so scary if you know there are automated integration tests which will tell you something is broken.

Other types of testing (functional tests, like testing user stories, or performance testing) seem to me like obviously separate phases of the development, and you can hardly argue they can be done implicitly while programming.

I run into this exact problem a surprising amount. People in various corners of the business use terms like "data" and "testing" ambiguously without themselves really understanding what they're asking when they utter these terms.

So when we're constructing some throw-away proof of concept, our development plan may not include any period of formal functional testing (because it's not necessary), or regression testing (because there's no prior version to compare with) or even writing regression tests (because there's not going to be a future version, either^†).

But of course this doesn't mean we never actually ran the code. Basic developer-level tests are an inherent part of programming. If you'd never "tested" what you wrote, you would literally be programming blind. At its most extreme, this approach would mean not even performing a check that your code compiles, and in my experience getting something to compile first time is very rare. So obviously you are going to be doing some amount of testing of your code before you announce that it works and is ready for demonstration.

But managers don't always get this. Your project manager will say "why did you spend time on testing? I said not to". And then when you decide "okay I won't count that as 'testing' because it was just basic development sanity checks" your line manager comes to you wondering why — and how — you haven't done any testing at all.

At this point I find it's best to qualify the word "testing" by replying with a sentence. Something like:

We haven't done a formal test run (and aren't planning to because it's a Proof of Concept), but my basic development-level tests show that it works well enough for the demo tomorrow.

It's worth noting that since your line manager asked you for a spreadsheet of test results, that suggests to me that you actually are expected to do some formal testing. In that case, he is actually disagreeing with the project manager and you should refer them to each other to figure out what they want your team to do.

^{† Hahaha yeah right. Proofs of concept always end up evolving into real products. Management always ruins things by making that happen. Nowadays I try not to cut corners even on a proof of concept, because that's the best way to give yourself a tremendous ton of technical debt on the first version of your product.}

Manual Testing is a large area, I'd break it down into:

• Unit testing
  Do basic functions achieve their purpose when used by the user?

• Integrated Testing
  Are related services and datastores updated and communicated with correctly?

• Performance Testing
  How long do responses take? How many users can be supported simultaneously (load)

• Exploratory Testing
  Does the system work from a real users perspective?

These are similar to the 4 quadrants of Agile testing, note that the tests could be manual or automated, the areas and need are similar.